[time-nuts] Raspberry Pi 4B Ethernet delay

[Fiorenzo Cattaneo]

Well, yes and no......... The speed of light is course always the same, but
in case of ethernet the latency is determined by the time it takes for the
packet to be transmitted at a given bitrate , plus switch delays, and output
/ input delays on the ethernet controller as well as the software stack
overhead, which dominates.

Suppose the average NTP packet is 100 bytes, and assume standard 8-10
encoding no with extra overhead, so 1 byte = 10 bits:
On a 100 Mbps network the latency would be 10 microseconds.
On a 1 Gbps network the latency would 1 microsecond.
On a 10 Gbps network 0.1 microseconds and finally on a 100 Gbps network 0.01
microseconds.

In practice the expected latency is in large part determined by the network
switches, network adapters, but most importantly software stack.

This is what I get using Intel 6154 3.0 Ghz XEON servers connected via state
of the art Arista switch (packets in this switch are forwarded from ingress
port to egress port immediately -- there is no store and forward in the
switch) 100 Gbps network:

* One way latency with ICMP ping at 100 bytes 100 Gbps network ping latency
: 65-80 microseconds
* One way latency with MPI messages at 128 bytes on the same 100 Gbps
network, this time using RDMA (direct data transfer done in hardware,
without any involvement from the part of the kernel software stack): 1.65
microseconds +/- 0.1 microsecond.

Even when using hardware assisted RDMA the best achievable latency is 1.65
microseconds, compared with expected network transit time of 0.01
microseconds.
With these numbers as reference, hardware imposed latency is about 10x the
network transit time at 100 Gbps, and software imposed latency is about 50x
the hardware overhead.

Note how even on this supercomputing hardware the ICMP one way ping latency
of 65 microseconds is only one order of magnitude better than the 254
microseconds repoted by David on the RPI 4.

So, by far, the network software stack imposes the largest overhead. A 1
Gbps network will be 10 times faster than a 100 Mbps network for data
transfer, but when it comes to latency, the benefits are minimal, although
you probably get some marginal benefits due to the fact that a 1 Gbps
network chip is more recent, hence probably a bit more optimized. In both
cases hardware UDP/IP offload has been around for a long time, and it's
probably not particularly relevant for 100 bytes packet size anyway.

My expectation here would be that RPI 4 to be better than RPI 3 because it
avoids the USB bus middle man, if not in terms of absolute latency,
certainly in terms of jitter.

In case anyone cares to know, the hardware I'm using is Oracle Cloud HPC
Cluster high-frequency offering with RDMA hardware. All the numbers I have
quoted from my own measurements are publicly available.

-----Original Message-----
From: time-nuts <time-nuts-bounces at lists.febo.com> On Behalf Of Achim Gratz
Sent: Friday, 8 November, 2019 09:44
To: time-nuts at lists.febo.com
Subject: Re: [time-nuts] Raspberry Pi 4B Ethernet delay

David J Taylor via time-nuts writes:
> I did some very simple tests to see how the RPi 4B with its better
> Ethernet connection compared with the earlier model B (rather old!).
> My test configuration isn't good, as there are a couple of switches
> between the two RPi cards and the monitoring PC:
>
>  https://www.satsignal.eu/ntp/BBB-vs-RPi.html#update
>
> but the summary is:  the model B reports 0.464 ms mean delay, and the
> model 4B reports 0.254 ms, so 0.21 ms better.

FWIW, that's roughly the same latency I see for a TinkerBoard running the
ethernet interface at 100MBit/s and the buffering window reduced to 0.67µs.
The latency would be somewhat lower with the interface run at 1GBit/s,
slightly below 200µs IIRC.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

_______________________________________________
time-nuts mailing list -- time-nuts at lists.febo.com To unsubscribe, go to
http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
and follow the instructions there.