[time-nuts] Raspberry Pi NTP server

[Trent Piepho]

On Fri, Jul 10, 2020 at 12:17 PM Tim S <tim.strommen at gmail.com> wrote:
>
> I believe the issue is arising from the understanding of "what" is being
> interrupted.
>
> When a UART or GPIO is triggering an interrupt - for the UART it's a FIFO
> notice that data is available to be pulled out, for the GPIO it's
> notification of a state change (and optionally, a specific state change:
> high-low/low-high).

On a processor like the Ti Sitara AM3358 found in the beaglebone or
the Broadcom processors in the various RPis, it's a little more
complex than that.  Both at the hardware level and at the software
level.

The UART IP block in the SoC has a module clock supplied from the
processor clock tree.  Likely two, one to drive the logic in the UART
and another to clock the external signals.  For the AM3358, there is a
100 MHz clock for the UART and an up to 48 MHz clock for the external
signals.

If the UART FIFO has reached a trigger point or the RTS line has
changed state, the UART will only raise its interrupt line on the edge
of the module clock, e.g. 100 MHz.  A GPIO controller would be the
same way.

That interrupt line does not go straight to the processor.  It goes to
an interrupt controller, such as the ARM GIC or a proprietary
controller.  The AM3358 uses a TI controller.  This controller deals
with priority of interrupts, masking, level/edge triggering (this is
about how the interrupt line from the UART or GPIO controller to the
INTC acts, not about how a GPIO will trigger an interrupt), steering
of interrupts to multiple CPU cores, and finally raising the interrupt
line in a CPU core.  Which might not happen immediately if another
interrupt is already executing.  The interrupt controller has its own
clock domain too, running at maybe 1/2 the processor's frequency.  It
takes a non-zero and non-constant amount of time for the INTC to pass
on the interrupt.

Once that happens, the CPU automatically does some stuff to respond to
the interrupt line.  It needs to finish executing the current CPU
instruction.  They don't all take one cycle, so this might be more or
less time.  The CPU also needs to jump to the main interrupt handler,
saving the current address.

We haven't even run one single software instruction yet, but we are
finally at the interrupt handler and have some control.  But it has
some stuff it must do first.  Save registers, switch stacks, etc.
That might access memory, which might not be cached, and so there's
more delay here.

It's got to figure out what interrupt occurred from the INTC.  This is
just a read from the memory mapped register.  But that means an AXI
bus transaction.  The CPU sends a request over the SoC interconnect
addressed to the interrupt controller.  The controller sends a
response back with the data.  Maybe there is contention on the AXI bus
due to data from other AXI masters, e.g. bus mastering DMA from the
USB controller.  More non-constant delays.

At this point the kernel interrupt handler will have figured out the
interrupt is from the UART and will call the handler registered by the
UART driver, GPIO driver or maybe a PPS driver.  What I've seen
discussed seems to treat this as the starting point of an interrupt.
But it's hardly the start.  We've gone through several IP blocks,
crossed a half dozen clock domains, sent and received messages via an
on chip network, and executed hundreds of instructions.

> When a USB PHY trips an interrupt it's to indicate that carrier data is
> ready, or for a USB MAC that burst blocks over the link are being decoded.
> There is an intermediate step between the USB carrier data being
> received/decoded, and being interpreted by the driver as a specific type of
> data - and the time that it is made available to the host OS.  Because that
> is handled in the driver, it is subject to all of the other host OS
> interrupts - so timing is not deterministic for the USB driver.

To figure out there was an interrupt from the UART, that it was due to
RTS changing, and do something about it, it takes quite a few
instructions.  And a number of register accesses that needs to do to
the UART, usually that means from over AXI to a AXI to APB interface
than then APB to a UART (I've never seen a UART as an AXI master, they
are always on a simpler peripheral bus like APB or OCP).  This takes
time.

Figuring out a USB interrupt was due to the completion of an interrupt
transfer, and that an interrupt is present in the end point, also
takes that same stuff.  Except the USB controller might be on AXI
directly.  It's quite a bit more complex.  But there isn't really a
fundamental difference on the software level.