System time is off by up to hundreds of milliseconds despite NTP sync before boot

I'm running a couple of servers which need a pretty tight time sync (<50ms) as they are running a Paxos algorithm. The servers are running NTP and are successfully sync at one point. According to hwclock the 11-minute mechanism is enabled, so the system time should be copied to hardware clock.

However, I see that after a reboot the system time can be off by as much as 300ms compared to the time just before a reboot. Is it unreasonable to think that after a reboot the time should be within 50ms of the time just before reboot?

My initial reaction was that 300ms seems like an awful lot, but I do have numbers to produce, and they show that @Law29 is right:

1. One of my machines over a normal week:
   • Frequency:
   • System peer offset:
2. Same system, shorter period with a reboot involved:
   • Frequency:
   • System peer offset:
   • Scatter plot of the peers

(Hope you can read all the numbers on the graphs OK - drop me a comment if not.)

As you can see, there's a rather large discrepancy. It surprised me how much it was, and also how long it took to get back on track with the frequency correction, considering that there's a stratum 1 GPS source on my local network. And given that the peer samples are fairly tightly clustered on the plot, it's clearly a problem with the local clock, not inconsistent network delay during startup. (For the record, the hardware is a Shuttle DS437 fanless mini-PC with a dual-core Celeron 1037U @ 1.8 GHz.)

So the takeaways are probably:

1. make sure ntpd is successfully writing the NTP drift file,
2. make sure the kernel's 11-minute timer to update the hardware clock is on (See "Automatic Hardware Clock Synchronization by the Kernel" in man hwclock for details), or your shutdown process is updating the hardware clock,
3. make sure ntpd has 4-10 reachable sources (in iburst mode), and
4. configure your startup dependencies so that ntpd has a chance to fix the clock before Paxos starts.

I do not have numbers to produce, but it seems probable that the interface used to set the clock at boot only has precision down to the second.

You do not state your OS, but on all Unix-like systems it is possible to insert a dependency on NTP time in the boot process.

The NTP daemon is started at boot, but often it immediately backgrounds itself and boot continues while the NTP daemon looks for servers to sync to -- this is so that boot is not delayed in case the machine is not connected to the network.

In this case, you will want to make sure that the ntp daemon is started in a way that will correct an offset by stepping at boot. This can be, for example, ntpd -gx or chronyc -q. You may also wish to insert a check that the offset is acceptable before starting your workload.