[time-nuts] Re: i211 timestamping (was: Network interface cards that support timestamping)

Wed Feb 22 22:23:10 UTC 2023

Dear time-nuts,

On Tuesday, 14 February 2023 01:50:04 CET, John Miller <john at millerjs.org> 
wrote:

> Hi JÃ¼rgen,
> Finally getting around to sinking my teeth into doing i210 PPS stuff, so I
> am finally revisiting this. 

Great, I'm happy to have some company in this project :-)

> So, first off - thank you for the very
> informative post - lots of good information in here. I am very interested
> in seeing your design data for a mini PCIe board with pogo pins for the SDP
> interface, thank you for offering. 

You got mail ;-)

> Compiling testptp with a static library worked, of course, and I'm a little
> bit disappointed in myself for not thinking of this - I'm pretty rusty at
> this, go figure. In any case, it seems to be working fine - most of the
> time. I can use it to list capabilities, list pin configuration, and
> set/get PTP time, no problem, and I can see that SDP0 is configured for
> "external time stamp". Sometimes when I try and run it to view timestamp
> events (e.g. ./testptp -e 10) I get a cryptic "device in use" type error
> but I'm not quite sure what is hanging it up.

To me that happens when I don't have the pin configured for timestamping 
correctly, i.e. forgotten to run "-L x,1" or configured the wrong ptp-device. 
Note: testptp requires a '-d' switch whereas phc_ctl does not.

> > ts2phc -c eth3 -s generic -l 7 -f /etc/linuxptp/ts2phc.conf -m -q
> 
> i assume the contents of ts2phc.conf are the "[global] ... " that you posted
> below it, is that correct?
yes.

As I am trying to learn the intricacies of the i211 chip and its PTP timing, 
maybe time-nuts find the following technical details useful:

I did some experiments where I programmed the SDP1 to output 1pps and fed it 
back via cables of calibrated length into SDP0 of the same i211 chip for 
timestamping:
# testptp -d /dev/ptp2 -i 0 -L 1,2 -p 1000000000

The i211-chip system timer is incremented by a fixed value each of its 125 MHz 
clock cycles. To set the increment fix to 8 ns, first the frequency-adjustment 
has to be set to zero: 
# phc_ctl eth2 freq 0

Then for an external cable delay of 12.8 ns (measured with VNA by reflection 
into the unpowered chip), my pulse ends up just between two detection clock 
cycles and I measure either a 24 ns or 32 ns delay for the detected edge. The 
jitter seems O.K.: Changing the cable delay length by 250 ps to either side 
leaves me firmly in the 24 ns or 32 ns-domain. 

# testptp -d /dev/ptp2 -i 0 -L 2,1 -e 10
set pin function okay
external time stamp request okay
event index 0 at 1677064396.000000032
event index 0 at 1677064396.500000032
event index 0 at 1677064397.000000024
event index 0 at 1677064397.500000032
event index 0 at 1677064398.000000032
event index 0 at 1677064398.500000024
event index 0 at 1677064399.000000032
event index 0 at 1677064399.500000032
event index 0 at 1677064400.000000024
event index 0 at 1677064400.500000032

That means, that there is a delay of at least a little bit more than two clock 
cycles between generation and time-tagging of the pulse (32ns - 12.8 ns = 19.2 
ns). Unfortunately, the data sheet of the i211 is silent on where this comes 
from - whether the output pulse is produced late, or whether the input pulse 
is registered late or whether it's both, leaving three possibilities:

1) pulse produced within the right clock cycle but tagged 2 clock cycles after 
arrival
2) pulse produced 1 clock cycle late and tagged 1 clock cycle after arrival
3) pulse produced 2 clock cycles delayed and tagged at the right clock cycle.  

My best guess is 1) (assuming a 2ff-synchronizer within the i211-chip to 
sample the asynchronous input signals) which the i211 designers just forgot 
about to document in the datasheet, but we really cannot know, leaving us with 
this 2.5-cycle systematic offset uncertainty between the internal timer and 
the external signals. 

Since the behavior is stable and predictable, this is only significant if time 
stamps with nanosecond precision leave or enter the computer on other ways 
than via the i211 SDP pins (e.g. another brand network interface, a DAC/ADC 
card, GPIO-pins etc.). 

Using my little interface board, I can now also lock two i211 interfaces on 
the same APU-motherboard (i.e. each with independent 25MHz crystals), either 
via ethernet & ptp4l or by feeding the 1pps-output of one chip into the other 
and using ts2phc. Attached you find some scope traces of the 1pps outputs of 
master (CH2, teal) and slave (CH3, pink). 

While the two crystals clocking the i211 drift slowly with respect to each 
other, the software can decide to correct the 'frequency' of the slave. As it 
is not really possible to change the real frequency of the hardware oscillator 
(there is no VCO involved) the software instead changes the amount the chip's 
system time is incremented at each clock cycle, i.e. the output pulse pulse 
timing can only be stepped by 8 ns increments effectively, leaving an at least 
one clock cycle wide pink shaded area in the scope traces that fundamentally 
cannot be avoided with a 8 ns timing detection resolution even with a perfect 
lock.

When connected directly via an Ethernet-cable the two interfaces lock nicely 
to each other via ptp4l (see ptp4l.png). When the master frequency tuning is 
set to 0, the slave is locked at an average offset within approximately -4 ns 
to +4 ns depending on the master clock-value modulo 8 ns.  If the master 
'frequency'-adjustment is unequal 0, the timing of the master 1pps is dithered 
with 8 ns steps, effectively doubling the jitter-variance between the 1pps 
pulses. 

When 1pps from eth3.SDP2 is fed into eth2.SDP3 (short connection with less 
than 1 ns propagation delay) and I lock via ts2phc, I get an 21.6 ns offset on 
average, agreeing roughly with the 19.2 ns-value obtained by time-tagging the 
pulses produced within the same chip. This average shift can be corrected for 
by the ts2phc.extts_correction parameter.

> Thanks very much for all your info, this is very exciting!
Indeed. As soon as I get some commercial PTP equipment, I'm curious to see 
whether the offsets agree...

Cheers,
	JÃ¼rgen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NewFile12.png
Type: image/png
Size: 49562 bytes
Desc: not available
URL: <http://febo.com/pipermail/time-nuts_lists.febo.com/attachments/20230222/7bc266f5/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NewFile14.png
Type: image/png
Size: 51901 bytes
Desc: not available
URL: <http://febo.com/pipermail/time-nuts_lists.febo.com/attachments/20230222/7bc266f5/attachment-0001.png>