If your Ethernet chipset (mac or phy) has timestamping capabilities, you can use Chrony which has hardware timestamp support. This greatly improves accuracy, and generally eliminates the CPU loading issue. Denny