[time-nuts] Re: NIST NTP servers way off for anyone else?

Tue Dec 14 17:24:35 UTC 2021

I see some high offsets on my local monitoring (synced to time.google.com).
Especially time-e-b on ipv6.

[image: nist-ntp-graph.png]

On Tue, Dec 14, 2021 at 5:26 PM Steven Sommars <stevesommarsntp at gmail.com>
wrote:

> The Gaithersburg servers are accurate.  This plot shows Gaithersburg
> time-e-g.nist.gov for the current month.
> [image: image.png]
> My monitoring client is located near Chicago and is Stratum-1 GPS sync'd.
> Typical round-trip time to Gaithersburg is ~27 msec.
> On 2021-12-07 a few monitoring polls saw RTT of ~100 msec.  This changes
> the computed offset from
> ~0 msec to to ~40 msec (  (100-27)/2. ).     Such transient increases are
> often called  "popcorn spikes"
> Many NTP clients including ntpd and chrony contain logic that identifies
> and suppresses these outliers
> Further, Gaithersburg is subject to fiber-cut outages and other planned &
> unplanned network outages.
> If you look carefully at the diagram, you can see a brief outage beginning
> at about 2021-12-08 03:00 UTC.
>
> I have other monitoring clients and can produce similar diagrams from other
> locations.  A monitoring client
> in Japan saw this for the same Gaithersburg server:
> [image: image.png]
> The delay spikes occur at different times and have different signs.  [The
> 2021-12-08 outage is still present]
> See   http://leapsecond.com/ntp/NTP_Paper_Sommars_PTTI2017.pdf for a
> discussion of why there are multiple
> offset bands.  In the same paper there are examples of sustained high
> delay, something that
> popcorn spike suppressors cannot eliminate.
>
> Magnus summarized the situation.  Either asymmetric network delay or a
> misbehaving NTP server can
> cause the computed offset to be non-zero.    The former is very common.
> NTP servers, even stratum 1's
> driven by GPS, are sometimes in error.
>
>
> Steve Sommars
>
>
>
>
> On Tue, Dec 14, 2021 at 4:21 AM Magnus Danielson via time-nuts <
> time-nuts at lists.febo.com> wrote:
>
> > Hi,
> >
> > On 2021-12-14 02:26, Adam Space wrote:
> > > I'm not sure if anyone else uses the NIST's NTP servers, but I've
> noticed
> > > that the offsets I'm getting from Gaithersburg servers seem to be
> > > really far off, like 40-50 ms off. This is pretty odd since they
> usually
> > > have a 2 - 3 ms accuracy at worst.
> > >
> > > It is interesting to think about what is going on here. NIST has a
> > secondary
> > > time scale
> > > <
> >
> https://www.nist.gov/pml/time-and-frequency-division/time-services/utcnist-time-scale/secondary-utcnist-time-scales-and
> > >
> > > at Gaithersburg, maintained by a couple of caesium clocks that are
> > > typically kept within 20ns of UTC(NIST), i.e. their primary time scale
> in
> > > Boulder. They also host their remote time transfer calibration service
> > and
> > > their Internet Time Service (i.e. NTP servers) out of Gaithersburg.
> > >
> > > It seems highly unlikely that their time scale there is that far off.
> One
> > > thing that immediately comes to mind is asymmetric network delays
> causing
> > > this. I do think this has to be the reason for the large discrepancy,
> but
> > > even so, it is an impressive feat of asymmetric path delays. The
> maximum
> > > error in offset from a client to server due to asymmetric network
> delays
> > is
> > > one half of the delay. (This corresponds to one path being
> instantaneous
> > > and the other path taking the entire delay time). When I query their
> > > servers, I am getting about a 45ms offset, and a delay of around 100ms.
> > > This would mean the maximum error due to asymmetric path delays is
> around
> > > 50ms--and less even if we're being realistic (one of the delays can't
> > > literally be zero). Basically, for the offset error to be accounted for
> > > primarily by asymmetric network delays, the delays would have to be
> > *very*
> > > asymmetric.
> >
> > For the asymmetry to be 45 ms, the difference between forward and
> > backward path would need to be 90 ms, since the time error will be the
> > difference in delay divided by 2. The round trip time is instead the sum
> > of the two delays.
> >
> > Now, as you observe this between two clocks with a time, difference, the
> > time-difference add onto the TE, but does not show up on the RTT.
> >
> > So, 90 ms difference would fit, but delays would be 95 ms and 5 ms +/- 5
> > ms, since we can trust the RTT to be unbiased. Now we come to what is
> > physical possible, and 5 ms is 1000 km fiber delay. You can calculate
> > yourself from your location the minimum distance and thus delay. In
> > practice fiber is pulled not as straight as one would wish. I use at
> > least square root of 2 as multiplier, but many agree that this is still
> > optimistic and it can be far worse.
> >
> > What can cause such delay in a network? In IP/MPLS, the routing
> > typically does not care about forward and backward direction being the
> > same. Rather, they trim it to shed the load, i.e. Traffic Engineering.
> > That means that for two pair of nodes in that network, it can be sent
> > over a shorter path in one direction and longer in the other. In
> > addition, buffer fill levels can be high on a path, meaning that you end
> > up in the end of a buffer for each router hop due to traffic load. Delay
> > is a means to throttle TCP down in rate. Random Early Discard (RED) is
> > meant to spread that evenly between streams to cause throttle earlier
> > than dropping packets due to full buffers, but it still means dropping
> > packets. That affects UDP traffic too. MPLS-TE then tries to work on
> > that on a secondary level.
> >
> > With that, depending on your actual distance, which I do not know, it
> > becomes fuzzy if the network or servers have asymmetry. If you have
> > enough distance, then some of the time error can not be allocated to
> > network asymmetry as the short path needs to be higher. This then needs
> > to be allocated over to clock errors.
> >
> > All this is a result of having three unknowns and two measures, you
> > cannot fully resolve that equation system. It needs aiding. Having the
> > right time on one end does not help if one attempts to know the time
> > error of the other end.
> >
> > It would help if you could add observations from other locations near
> > Gaithersburg, network wise.
> >
> > > Is anyone else experiencing the same thing?
> >
> > Which makes this question very relevant. Measuring with less of the
> > biases and noise of the network may provide clearer answer on the
> > Gaithersburg servers.
> >
> > Cheers,
> > Magnus
> > _______________________________________________
> > time-nuts mailing list -- time-nuts at lists.febo.com -- To unsubscribe
> send
> > an email to time-nuts-leave at lists.febo.com
> > To unsubscribe, go to and follow the instructions there.
> >
> _______________________________________________
> time-nuts mailing list -- time-nuts at lists.febo.com -- To unsubscribe send
> an email to time-nuts-leave at lists.febo.com
> To unsubscribe, go to and follow the instructions there.

-- 
Michael Rothwell
michael at rothwell.us
(828) 649-ROTH
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nist-ntp-graph.png
Type: image/png
Size: 1614510 bytes
Desc: not available
URL: <http://febo.com/pipermail/time-nuts_lists.febo.com/attachments/20211214/1524d8f4/attachment.png>