[time-nuts] Raspberry Pi NTP server

[Tim S]

I'll make a penultimate comment here.

As is pointed out USB is good at speed, and plug-and-play - what it is not
good at is deterministic time.  In general the main-line Linux is also in
the same boat.

Put another way, one can transmit 100's of megabits of data over USB 2.0,
but guaranteeing >>when<< that'll happen is a different story.  USB handles
that by moving giant chunks of data at a single event, rather than sending
one byte at a time at a specified rate.  Using what USB calls bulk
transfer, a block of data is written to a FIFO, then when the host machine
is ready to serve the USB driver's interrupt, it'll empty the whole buffer
during that read, so that it can go service other interrupts.  The actual
processing of that block data may then be subject to queued handlers at an
even more non-deterministic rate.  In fact, I know from similar Windows
experience, that it can be user space (lowest "system" priority) - where
it's first come first serve.

As a user, most of us are used to user-space timing.  Generally at 60Hz
frame-rate (1sec/60 = 16.666mSecond) if it happens within a frame or two a
computer seems "responsive" or "fast".  Likewise in gaming, the normal
closed loop from perception to physical response is on the order of 1/8 -
1/4 second.  For a 1PPS edge, that is a literal eternity.

If the previously discussed Linux patch for real-time operation is used
(for example "preempt-RT", which a significant effort is being made to push
into the mainline kernel, see: https://www.youtube.com/watch?v=BxJm-Ujipcg )
one can get stable latencies at or under 10uSec.  This is assuming some
caveats:  1) The time critical task is handled by a dedicated core (i.e.
setting the process "affinity" to a single core),  2) ensuring that no
other time critical tasks get assigned to that core (load/unload cycles add
some unpredictability to latency), and 3) the code handled during the
time-critical event is not so long that it does not complete before the
next event occurs.   Tuning Linux to sub 10uSec latencies is possible, but
at the moment it is in my opinion still a bit of "art" - so be prepared for
a measure of effort to achieve the tightest control.

It's important to understand that when people are talking about
"real-time", they don't necessarily mean that it's fast - it means that the
time understood to have elapsed is real/understood/controlled.  When an
event happend or is intended to have happened, is when it actually occurred
(aka: low jitter, in common time nut vernacular).

Another resource you might consider taking a look into is LinuxCNC - where
consistent timing and low latency have a direct impact on how round a curve
is, or whether a machine crashes or not.

I understand that a laptop may be convenient - but I'd offer that a purpose
built RaspberryPi with a dedicated high-timing accuracy GPS receiver tied
to UART and a GPIO are likely to get you closer.  And by hand tuning the OS
implementation, considering switching from NTP to the leaner/refactored
NTPSec ( https://www.ntpsec.org/ ), and perhaps looking at some of the
efforts towards getting PCIe out of a RaspberryPi 4 (
https://www.hackster.io/news/pci-express-on-the-raspberry-pi-4-9b03c59f7a04
 and
https://hackaday.com/2020/07/01/adding-pcie-to-your-raspberry-pi-4-the-easier-way/
)
- one can add on a dedicated PCIe NIC with IEEE1588 support (something like
a second-hand Mellanox ConnectX-3 Pro, p/n: MCX311A-XCCT) -
because remember the RPi NIC is over USB also.

It's a bit of work for certain - but it can be packaged into a project box
once, and then you just throw that into a suitcase rather than having to
worry about how a laptop is architected any time you have to buy
a new one.  Having discussed this, and watched the discussion for a few
days now - I'm inspired to build one.

-Tim


> Date: Sat, 11 Jul 2020 18:57:39 -0700
> From: jimlux <jimlux at earthlink.net>
> To: time-nuts at lists.febo.com
> Subject: Re: [time-nuts] Raspberry Pi NTP server
> Message-ID: <8a28aabb-e419-190f-37a6-e1a1476378da at earthlink.net>
> Content-Type: text/plain; charset=utf-8; format=flowed
>
> ...
> I would not expect another kind of USB to serial converter to do better.
> The problem is higher up in the way that Linux handles USB devices. The
> USB hardware can certainly handle higher rates (and does), but the
> "software interrupts" as the event travels up the stack limits the
> timing resolution.
>
> You might want to look into one of the "real time" linux kernels or
> other similar implementations - they might have "turned some of the
> knobs" to improve the handling of device data.
>
> USB device handling in Linux (and Windows, and Mac OSX) is quite
> complex, if only because USB itself is quite complex in that it has to
> support multiple "kinds" of devices with wildly varying properties (HID,
> Mass Storage, Isochronous data, etc.) - Not to mention all the
> complexities associated with hot plugging and unplugging and
> "enumeration" and "power control".
>
> You might also want to delve into the handling of USB request Blocks
> (URBs) which is how Linux handles USB related events.
>
>
> https://www.oreilly.com/library/view/linux-device-drivers/0596005903/ch13.html
>
> https://www.kernel.org/doc/html/v4.12/driver-api/usb/index.html
>
> The above document describes a variety of ways to get at USB devices in
> non-standard ways, through the USB API.
> ...
>
> Date: Sun, 12 Jul 2020 04:10:45 -0700
> From: Trent Piepho <tpiepho at gmail.com>
> To: Discussion of precise time and frequency measurement
>         <time-nuts at lists.febo.com>
> Subject: Re: [time-nuts] Raspberry Pi NTP server
> Message-ID:
>         <
> CA+7tXii33Oo_WtZ_E0jTyKNTv9yhh66A-JALJGTM0d+fegq0kg at mail.gmail.com>
> Content-Type: text/plain; charset="UTF-8"
>
> On Sat, Jul 11, 2020 at 7:37 PM jimlux <jimlux at earthlink.net> wrote:
> >
> > You might want to look into one of the "real time" linux kernels or
> > other similar implementations - they might have "turned some of the
> > knobs" to improve the handling of device data.
>
> The real time kernel primarily is about trying to get closer to making
> hard real time guarantees.  To put a hard maximum on latencies.  You
> can't really do that on Linux, but the real time patches made it
> better.  Mostly this is about finding a place where X could not happen
> in time because the CPU was doing Y instead, and Y couldn't be
> interrupted.  And find a way to interrupt Y so X doesn't have to wait.
>
> One of the biggest changes was to thread all interrupts.  So that
> interrupts can be interrupted.  If you are after low interrupt
> latency, this actually makes things worse.  But it makes the worst
> case much better than before, at the expense of the average case.
> ...
>