←back to thread

How Precision Time Protocol is being deployed at Meta

(engineering.fb.com)
221 points mfiguiere | 2 comments | 21 Nov 22 15:41 UTC | HN request time: 0.51s | source
Show context
amluto ◴[21 Nov 22 23:24 UTC] No.33699625[source]▶
I’m wondering whether the extremely careful GNSS part is really needed. A microsecond of offset between two servers in the same datacenter could easily matter, but I suspect that, if an entire datacenter were off by a microsecond, everything would be fine — communicating from that datacenter to anywhere else will take well over a microsecond, so an offset of this type would be a bit like the datacenter wiggling around in space a bit.

On a different note, there’s an Intel feature called the Always Running Timer. In theory one ought to be able to directly determine the TSC <-> NIC clock offset using the ART. I’m not sure anyone has gotten this to work, though.

replies(2): >>33699672 #>>33700902 #
1. error503 ◴[22 Nov 22 02:08 UTC] No.33700902[source]▶
Accurate delay compensation is necessary to enable redundancy. IF you need multiple GrandMasters at different locations in the facility, using independent RFoF systems and antennas, they will have different GNSS delays, and that difference will propagate down into uncertainty on the hosts. There are other ways to eliminate this uncertainty than to characterize the full delay, but if the racks are on opposite ends of a giant data centre, that might be as or more difficult than just going through those motions.

If you just have one GM, then sure, the delay means you will have a larger fixed offset from TAI/UTC, but that won't be consequential, and you'll still get the benefits of a tightly synchronized monotonic clock. Until that GM fails, and it all goes haywire.

replies(1): >>33716196 #
2. bradknowles ◴[23 Nov 22 07:22 UTC] No.33716196[source]▶
It's a hard problem to solve. You end up doing something like the NIST TMAS service (see https://www.nist.gov/programs-projects/time-measurement-and-...) using differential common view measurements to create a "Multi-Source Common-View Disciplined Clock".