How Precision Time Protocol is being deployed at Meta

Disclaimer: I work in finance, and have for 15+ years.

> Given the special hardware needed for the grand master clock to get down to nanosecond time scales, I'm doubtful this will be used in most data centres of most corporate networks.

The "special hardware" is often just a gps antenna and a PCI card though. In fact, many tier 1 datacenters actually provide a "service" where they'll either cross connect you directly to a PPS feed from a tier 0 grandmaster time service or plug your server into a gps antenna up on the roof. It isn't really that exotic. For financial application, especially trading ones, syncing a LAN timesync to a handful of nanoseconds is doable and optimal.

It is just a matter of time before non-finance sees reasons that better timesync is useful. Precision Time Protocol aka IEEE 1588 was released in 2002 and IEEE 1588 version 2 was released in 2008. This isn't exactly a new thing.

With the right hardware and a tier 0 timesource, modern ntp on modern hardware with modern networks can keep a LAN in sync subsecond. However, as a protocol, NTP only guarantees 1 second accuracy.

Disclaimer: I've been involved in supporting the NTP Public Services Project since 2003.

I assure you, with the right hardware and paying attention to your latencies, NTP can get you down below one millisecond accuracy. Poul Henning Kamp was doing nanosecond level accuracy with NTP back in the mid-aughts, but then he had rewritten the NTP server code, the NTP client code, and the kernel on the server.

As an NTP service provider, what you really want to keep an eye on is the Clock Error Bound that gives you the worst case estimate for how bad the time is that you could be serving to your customers. For the client side, you mainly care about just the accuracy you're actually getting.

> However, as a protocol, NTP only guarantees 1 second accuracy.

The "one second" number is not inherent to the protocol, but comes from a survey from 1999:

> A recent survey[2] suggests that 90% of the NTP servers have network delays below 100ms, and about 99% are synchronized within one second to the synchronization peer.

* http://www.ntp.org/ntpfaq/NTP-s-algo.htm#Q-ACCURATE-CLOCK

A 2005 survey found (AFAICT, see Figure 1) that north of 99% — more like 99.5% — of servers had offsets less than 100ms, and that 90% have offsets less than 10ms:

* PDF: https://web.archive.org/web/20081221080840/http://www.ntpsur...

The output of "chronyc tracking" from a randomly-selected system I help run:

    Reference ID    : […]
    Stratum         : 3
    Ref time (UTC)  : Wed Nov 23 13:35:21 2022
    System time     : 0.000002993 seconds fast of NTP time
    Last offset     : +0.000003275 seconds
    RMS offset      : 0.000008091 seconds
    Frequency       : 13.191 ppm slow
    Residual freq   : +0.001 ppm
    Skew            : 0.017 ppm
    Root delay      : 0.001615164 seconds
    Root dispersion : 0.000048552 seconds
    Update interval : 65.3 seconds
    Leap status     : Normal

So chrony's algorithm thinks that the system time is currently off by 0.000 002 993 seconds (3e-6), and on average it is off by 0.000 008 091 seconds (8e-6).

All I had to do to achieve this was have some infra nodes (that do other things as well) point to *.pool.ntp.org, and then have my cattle and pets point their NTP software to those infra nodes. No special hardware, no special software, no configuring of network switches and routers: just a few lines of Ansible and I get (estimated) microsecond (1e-6) error levels.

Yes, I've seen it get down to a few milliseconds of sync on the right hardware (boundary clock on the switches, stratum 0 timeserver with pps, etc), but the protocol only guarantees 1 second of sync. Am I incorrect in that assertion?