←back to thread

How Precision Time Protocol is being deployed at Meta

(engineering.fb.com)
221 points mfiguiere | 3 comments | 21 Nov 22 15:41 UTC | HN request time: 0s | source
Show context
throw0101a ◴[21 Nov 22 18:49 UTC] No.33696297[source]▶
So they state:

> One could argue that we don’t really need PTP for that. NTP will do just fine. Well, we thought that too. But experiments we ran comparing our state-of-the-art NTP implementation and an early version of PTP showed a roughly 100x performance difference:

While I'm not necessarily against more accuracy/precision, what problems specifically are experiencing? They do mention some use cases of course:

> There are several additional use cases, including event tracing, cache invalidation, privacy violation detection improvements, latency compensation in the metaverse, and simultaneous execution in AI, many of which will greatly reduce hardware capacity requirements. This will keep us busy for years ahead.

But given that NTP (either ntpd or chrony) tends to give me an estimated error of around (tens of) 1e-6 seconds, and PTP can get down to 1e-9 seconds, I'm not sure how many data centre applications need that level of accuracy.

> We believe PTP will become the standard for keeping time in computer networks in the coming decades.

Given the special hardware needed for the grand master clock to get down to nanosecond time scales, I'm doubtful this will be used in most data centres of most corporate networks. Adm. Grace Hopper elegantly illustrates 'how long' a nanosecond is:

* https://www.youtube.com/watch?v=9eyFDBPk4Yw

How many things need to worry the latency of signal travelling ~300mm?

replies(15): >>33696382 #>>33696446 #>>33696532 #>>33696586 #>>33697400 #>>33697855 #>>33698277 #>>33699184 #>>33700548 #>>33700694 #>>33700741 #>>33702033 #>>33702269 #>>33702897 #>>33705135 #
1. akira2501 ◴[22 Nov 22 05:17 UTC] No.33702269[source]▶
> But given that NTP (either ntpd or chrony) tends to give me an estimated error of around (tens of) 1e-6 seconds

Is that a hard bound or an average? If it's an average, then what are the limits of the bounds, both in magnitude and duration?

> and PTP can get down to 1e-9 seconds

We use it for audio, and the reason it works well there is because there is no exponential backoff with your peers, allowing for even small timing slips to become enough to notice. 1ms of latency is far too much for our application, we typically aim for 0.25ms.. and we're only running 96kHz. If we lose PTP sync, we notice within a few minutes.

Another advantage of PTP is it can operate as a broadcast and, as the article notes, switches can be PTP aware and help update the timing as the broadcast flows through the network. Conveniently, PTP also allows for multiple timing domains and masters to co-exist on the same network.

It's also an absurdly simple protocol, you can build a receiver for it in about 200 lines of C code. I've actually become quite taken with it since it was forced into our space about 10 years ago.

replies(2): >>33703958 #>>33705569 #
2. hoseja ◴[22 Nov 22 09:52 UTC] No.33703958[source]▶
I wouldn't call it absurdly simple.
3. throw0101a ◴[22 Nov 22 13:33 UTC] No.33705569[source]▶
> Another advantage of PTP is it can operate as a broadcast

NTP can work via broadcast and multicast.