[time-nuts] Re: NIST NTP servers way off for anyone else?

Tue Dec 14 16:26:28 UTC 2021

The Gaithersburg servers are accurate.  This plot shows Gaithersburg
time-e-g.nist.gov for the current month.
[image: image.png]
My monitoring client is located near Chicago and is Stratum-1 GPS sync'd.
Typical round-trip time to Gaithersburg is ~27 msec.
On 2021-12-07 a few monitoring polls saw RTT of ~100 msec.  This changes
the computed offset from
~0 msec to to ~40 msec (  (100-27)/2. ).     Such transient increases are
often called  "popcorn spikes"
Many NTP clients including ntpd and chrony contain logic that identifies
and suppresses these outliers
Further, Gaithersburg is subject to fiber-cut outages and other planned &
unplanned network outages.
If you look carefully at the diagram, you can see a brief outage beginning
at about 2021-12-08 03:00 UTC.

I have other monitoring clients and can produce similar diagrams from other
locations.  A monitoring client
in Japan saw this for the same Gaithersburg server:
[image: image.png]
The delay spikes occur at different times and have different signs.  [The
2021-12-08 outage is still present]
See   http://leapsecond.com/ntp/NTP_Paper_Sommars_PTTI2017.pdf for a
discussion of why there are multiple
offset bands.  In the same paper there are examples of sustained high
delay, something that
popcorn spike suppressors cannot eliminate.

Magnus summarized the situation.  Either asymmetric network delay or a
misbehaving NTP server can
cause the computed offset to be non-zero.    The former is very common.
NTP servers, even stratum 1's
driven by GPS, are sometimes in error.

Steve Sommars

On Tue, Dec 14, 2021 at 4:21 AM Magnus Danielson via time-nuts <
time-nuts at lists.febo.com> wrote:

> Hi,
>
> On 2021-12-14 02:26, Adam Space wrote:
> > I'm not sure if anyone else uses the NIST's NTP servers, but I've noticed
> > that the offsets I'm getting from Gaithersburg servers seem to be
> > really far off, like 40-50 ms off. This is pretty odd since they usually
> > have a 2 - 3 ms accuracy at worst.
> >
> > It is interesting to think about what is going on here. NIST has a
> secondary
> > time scale
> > <
> https://www.nist.gov/pml/time-and-frequency-division/time-services/utcnist-time-scale/secondary-utcnist-time-scales-and
> >
> > at Gaithersburg, maintained by a couple of caesium clocks that are
> > typically kept within 20ns of UTC(NIST), i.e. their primary time scale in
> > Boulder. They also host their remote time transfer calibration service
> and
> > their Internet Time Service (i.e. NTP servers) out of Gaithersburg.
> >
> > It seems highly unlikely that their time scale there is that far off. One
> > thing that immediately comes to mind is asymmetric network delays causing
> > this. I do think this has to be the reason for the large discrepancy, but
> > even so, it is an impressive feat of asymmetric path delays. The maximum
> > error in offset from a client to server due to asymmetric network delays
> is
> > one half of the delay. (This corresponds to one path being instantaneous
> > and the other path taking the entire delay time). When I query their
> > servers, I am getting about a 45ms offset, and a delay of around 100ms.
> > This would mean the maximum error due to asymmetric path delays is around
> > 50ms--and less even if we're being realistic (one of the delays can't
> > literally be zero). Basically, for the offset error to be accounted for
> > primarily by asymmetric network delays, the delays would have to be
> *very*
> > asymmetric.
>
> For the asymmetry to be 45 ms, the difference between forward and
> backward path would need to be 90 ms, since the time error will be the
> difference in delay divided by 2. The round trip time is instead the sum
> of the two delays.
>
> Now, as you observe this between two clocks with a time, difference, the
> time-difference add onto the TE, but does not show up on the RTT.
>
> So, 90 ms difference would fit, but delays would be 95 ms and 5 ms +/- 5
> ms, since we can trust the RTT to be unbiased. Now we come to what is
> physical possible, and 5 ms is 1000 km fiber delay. You can calculate
> yourself from your location the minimum distance and thus delay. In
> practice fiber is pulled not as straight as one would wish. I use at
> least square root of 2 as multiplier, but many agree that this is still
> optimistic and it can be far worse.
>
> What can cause such delay in a network? In IP/MPLS, the routing
> typically does not care about forward and backward direction being the
> same. Rather, they trim it to shed the load, i.e. Traffic Engineering.
> That means that for two pair of nodes in that network, it can be sent
> over a shorter path in one direction and longer in the other. In
> addition, buffer fill levels can be high on a path, meaning that you end
> up in the end of a buffer for each router hop due to traffic load. Delay
> is a means to throttle TCP down in rate. Random Early Discard (RED) is
> meant to spread that evenly between streams to cause throttle earlier
> than dropping packets due to full buffers, but it still means dropping
> packets. That affects UDP traffic too. MPLS-TE then tries to work on
> that on a secondary level.
>
> With that, depending on your actual distance, which I do not know, it
> becomes fuzzy if the network or servers have asymmetry. If you have
> enough distance, then some of the time error can not be allocated to
> network asymmetry as the short path needs to be higher. This then needs
> to be allocated over to clock errors.
>
> All this is a result of having three unknowns and two measures, you
> cannot fully resolve that equation system. It needs aiding. Having the
> right time on one end does not help if one attempts to know the time
> error of the other end.
>
> It would help if you could add observations from other locations near
> Gaithersburg, network wise.
>
> > Is anyone else experiencing the same thing?
>
> Which makes this question very relevant. Measuring with less of the
> biases and noise of the network may provide clearer answer on the
> Gaithersburg servers.
>
> Cheers,
> Magnus
> _______________________________________________
> time-nuts mailing list -- time-nuts at lists.febo.com -- To unsubscribe send
> an email to time-nuts-leave at lists.febo.com
> To unsubscribe, go to and follow the instructions there.
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 32489 bytes
Desc: not available
URL: <http://febo.com/pipermail/time-nuts_lists.febo.com/attachments/20211214/980a27f3/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 33496 bytes
Desc: not available
URL: <http://febo.com/pipermail/time-nuts_lists.febo.com/attachments/20211214/980a27f3/attachment-0001.png>