This is incredible. If I knew I'd be working on things like this (as opposed to optimizing ad revenue or "user engagement" by A/B testing button copy), I'd possibly consider working at Meta.
replies(5):
You need both.
PTP synchronizes clocks to a ridiculous precision, like down to nanoseconds. To do that, it uses support in the ethernet hardware. Hardware adds precision stamps to the PTP-related ethernet frames, so that the time calculations are free of jitters induced by the higher network stacks.
A cool thing is that PTP-aware routers/switches can actually rewrite the timestamps in PTP packets to subtract their own propagation delay accurately.
Something in the network has to use NTP, if it is important for the devices to have the correct calendar date and time; you don't use PTP over the Internet to get time from a remote server.
The real-time clock being synchronized isn't even the system clock in the first place, but a high resolution RTC that is in the network adapters and switches.
As an additional utility on top of the PTP stack, there is the possibility of synchronizing the host system's clock with the help of PTP.
PTP is used for things like generating precise signals in certain applications. E.g. Some devices that have PTP support can be programmed to generate periodic pulses which are clocked by the sychronized real-time clock. With PTP, two such signals from separate devices on the network can align very precisely. Such an application doesn't care about the time of day, just that the signals are synced.
> One could argue that we don’t really need PTP for that. NTP will do just fine. Well, we thought that too. But experiments we ran comparing our state-of-the-art NTP implementation and an early version of PTP showed a roughly 100x performance difference:
While I'm not necessarily against more accuracy/precision, what problems specifically are experiencing? They do mention some use cases of course:
> There are several additional use cases, including event tracing, cache invalidation, privacy violation detection improvements, latency compensation in the metaverse, and simultaneous execution in AI, many of which will greatly reduce hardware capacity requirements. This will keep us busy for years ahead.
But given that NTP (either ntpd or chrony) tends to give me an estimated error of around (tens of) 1e-6 seconds, and PTP can get down to 1e-9 seconds, I'm not sure how many data centre applications need that level of accuracy.
> We believe PTP will become the standard for keeping time in computer networks in the coming decades.
Given the special hardware needed for the grand master clock to get down to nanosecond time scales, I'm doubtful this will be used in most data centres of most corporate networks. Adm. Grace Hopper elegantly illustrates 'how long' a nanosecond is:
* https://www.youtube.com/watch?v=9eyFDBPk4Yw
How many things need to worry the latency of signal travelling ~300mm?
See “Commoditize Your Complement” - https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ , https://www.gwern.net/Complement
Is it really their "special sauce" though? Do these types of releases actually give away how these companies make money?
In this particular case, telling the world how to get to nanosecond levels of timekeeping doesn't really help any competitors take away Metabook's revenues or profits.
I think usually it's for company PR for various purposes (counteract bad press, attract new hires, etc.).
Sometimes to generate a bigger hiring pool that knows the stuff you're releasing. (And the open source story about crowdsourcing contributions, which sometimes might be worth the costs.)
I've also seen it around partnerships and customer collaborations and competition. Including to "commoditize your complement", or to kill one thing with what they'd rather use. (And, in industry/tech standards, corporate representatives often have motivation to try to bias the standard to their employer.)
In some cases, it's for individual employees' careers. Think how academic and some R&D jobs want research publications, or how some companies want people who do "talks".
Sometimes also for getting code/docs public, so employees can still use it when they leave.
It's quite likely we're entering a period where the current baseline performance of core infrastructure will be considered "good enough" and companies won't employ people to work on these general improvements.
Some of the justification for Google’s TrueTime is found in the Spanner docs:
https://cloud.google.com/spanner/docs/true-time-external-con...
Basically, you want to be able to do a “snapshot read” of the database rather than acquiring a lock (for reasons which should be apparent). The snapshot read is based on a monotonic clock. You can get much better performance out of your monotonic clock if all of your machines have very accurate clocks. When you write to the database, you can add a timestamp to the operation, but you may have to introduce a delay to account for the worst-case error in the clock you used to generate the timestamp.
More accurate timestamps -> less delay. From my understanding, less delay -> servers have more capacity -> buy fewer servers -> save millions of dollars -> use savings to pay for salaries of people who figured out how to make super precise timestamps and still come out ahead.
This kind of engineering effort makes sense at companies like Google and Meta because these companies spend such a large amount of money on computer resources to begin with.
https://github.com/grpc/grpc/blob/master/src/core/lib/iomgr/...
Very cool!
2) it serves as a venue for attracting other talented candidates who are likewise minded on working on technical problems.
3) when I was employed of FB it was a relatively flat hierarchy, which is to say there weren’t that many higher ups to convince that this should be done.
Arguably every program? The slowest part of modern programs is memory access. L1 cache memory access is ~1 nanosecond and RAM is ~50 nanoseconds.
Is 49 nanoseconds a lot? No, if you do it once. Yes, if every line of code pays the price.
Does that really matter in the end? If you're against writing code for ads/optimizing for quick dopamine hits, is it really so different to be writing the code that shows the ads vs the code that sets up the infrastructure to serve the ads? Or even infrastructure to support the developers who write the code for the ads?
If you don't want to support the ad-driven internet, don't work for companies who's entire business model is extracting as much data from users as possible in order to show the "right" ads.
And lots of this stuff is NOT secret sauce, it's basic business building-blocks that they need. It's not the advertising formulas.
I'm sure FAANG is very VERY happy that they can just run Linux everywhere and don't have to pay Sun or Microsoft a massive per-CPU fee for everything they do.
I want to contribute to building cool things, and building a really cool component becomes much less inspiring if it's only used to steer nuclear missiles. (dramatic but not unreasonable example)
I'm completely ignorant on the topic in general though... so I'm probably missing something. =)
In Meta's case, they fired a lot of "boot campers" as well, some only a few days into their job and before they had a team. Some returning interns even had their offers rescinded.
Not to sound cynical, but this article feels more like "let us release this to show impact and keep our jobs." Or damage control for their engineering hiring image.
Far for being absurd it seems a very straightforward extension of BSD socket API.
Lots of these companies opened up the ability for internal engineers to write tech blogs for the purposes of recruiting since the FAANGs were all in competition with one another. Presumably, the VPs haven't gotten around to closing the pipeline yet.
Or you know, just don't work for companies who tend to collect as much user data as they can?
offtopic but this sentence is beautiful to me.
So as user, this level of time synchronization seems to be many orders of magnitude of over kill. I'm sure at some level it is actually important, but it seems like the database sync or whatever is going on with notifications is woefully lagging.
On a different note, there’s an Intel feature called the Always Running Timer. In theory one ought to be able to directly determine the TSC <-> NIC clock offset using the ART. I’m not sure anyone has gotten this to work, though.
Or for use in a case where the network has a total absence of connectivity to any internet-based NTP sources (maybe because your management network doesn't talk to the internet at all, for many good reasons), and in the event of loss of transport connectivity to your own somewhere-in-the-region NTP servers in your management system, you want to be absolutely sure the system clocks on your local radio transport equipment, DWDM equipment, metro ethernet stuff are all precise.
Using receive-only GPS data as a master time reference source is effectively saying "we think that getting the time as it's set by the USAF/Space Force people at Schriever AFB, who run the GPS system, should be treated as our master point of reference and all other clocks should derive from that". It's not a bad policy, as such things go, because GPS maintaining extremely accurate time is of great interest to the US federal government as a whole.
Even a fairly small LTE cellular rooftop site, monopole or tower often has a similar receiver. It doesn't add a lot of cost.
An interesting movie might be a dystopian future where Facebook itself falls out of use but some type of rogue group takes over the Facebook datacenter infrastructure and uses it for their own purposes.
Might as well get ps level precision with sub-ns accuracy
For that use case, 1ns of inaccuracy is about 10ft all told (IE accounting for all the inaccuracy it generates).
It can be less these days, especially if not just literally using GPS (IE a phone with other forms of reckoning, using more than just GPS satellites, etc). You can get closer to the 1ns = 1ft type inaccuracy.
But if you are a cell tower trying to beamform or something, you really want to be within a few ns, and without PTP that requires direct satellite access or somet other sync mechanism.
Second, I'm not sure what you mean by special. Expense is dictated mostly by holdover and not protocol. It is true some folks gouge heavily on PTP add-ons (orolia, i'm looking at you), but you can ignore them if you want. Linux can do fine PTP over most commodity 10G cards because they have HW support for it. 1G cards are more hit or miss.
For dedicated devices: Here's a reasonable grandmaster that will keep time to GPS(/etc) with a disciplined OCXO, and easily gets within 40ns of GPS and a much higher end reference clock i have. https://timemachinescorp.com/product/gps-ntpptp-network-time...
It's usually within 10ns. 40ns is just the max error ever in the past 3 years.
Doing PTP, the machines stay within a few NS of this master.
If you need better, yes it can get a bit expensive, but honestly, there are really good OCXO out there now with very low phase noise that can more accurately stay disciplined against GPS.
Now, if you need real holdover for PTP (IE , yes you will probably have to go with rubidium, but even that is not as expensive as it was.
Also, higher end DOCXO have nearly the same performance these days, and are better in the presence of any temperature variation.
As for me, i was playing with synchronizing real-time motion of fast moving machines that are a few hundred feet apart for various reasons. For this sort of application, 100us is a lot of lag.
I would agree this is a pretty uncommon use case, and I could have achieved it through other means, this was more playing around.
AFAIK, the main use of accurate time at this level is cell towers/etc, which have good reasons to want it.
I believe there are also some synchronization applications that have need of severe accuracy (synchronous sound wave generation/etc) but no direct access to satellite signal (IE underwater arrays).
https://www.mouser.com/ProductDetail/HUBER%2bSUHNER/Direct-G...
Admittedly, at their scale, this is peanuts. But I wouldn’t buy one of these for a scrappy startup :). SparkFun will sell a perfectly serviceable kit for a few hundred dollars.
(If you are worried about lightning, then GPSoF looks like cheap insurance.)
If you just have one GM, then sure, the delay means you will have a larger fixed offset from TAI/UTC, but that won't be consequential, and you'll still get the benefits of a tightly synchronized monotonic clock. Until that GM fails, and it all goes haywire.
While I was hyperbolic in my pronunciation, I wasn't wrong in my data. It's been more than 5 years since those conversations. Glad someone got it done. Took 'em far too long.
From my understanding, basically this new iteration is better then their previous generation chrony based one because the uncertainty in the One Way Delay calculation is largely removed by having those Transparent Clocks capable of reporting their queuing delays. Basically the asymmetry of the delays are gone?
Personally, I don't think Chrony should be considered as a NTP implementation as clearly it can utilize those PTP hardware timestamps as well and those hardware timestamps are the "secret sauce" of PTP's high accuracy. With those PTP enabled switches, together with the fact that Chrony can already send NTP packages as PTP packages, surely Chrony can leverage such new capabilities as well with some reasonable updates.
they mentioned that most modern NICs have hardware timestamp capabilities, that is consistent with what I saw. The question is what is the availability of those PTP ready & enabled routers/switches capable of acting as a Transparent Clock? I have two concerns, most of existing routers/switches are not PTP ready, even for those ones with PTP capabilities, its PTP related features may not be enabled in production. Any experience or numbers to share? thanks!
The vast majority of people will probably end up working on product-related team compared to infrastructure-related teams.
> Given the special hardware needed for the grand master clock to get down to nanosecond time scales, I'm doubtful this will be used in most data centres of most corporate networks.
The "special hardware" is often just a gps antenna and a PCI card though. In fact, many tier 1 datacenters actually provide a "service" where they'll either cross connect you directly to a PPS feed from a tier 0 grandmaster time service or plug your server into a gps antenna up on the roof. It isn't really that exotic. For financial application, especially trading ones, syncing a LAN timesync to a handful of nanoseconds is doable and optimal.
It is just a matter of time before non-finance sees reasons that better timesync is useful. Precision Time Protocol aka IEEE 1588 was released in 2002 and IEEE 1588 version 2 was released in 2008. This isn't exactly a new thing.
With the right hardware and a tier 0 timesource, modern ntp on modern hardware with modern networks can keep a LAN in sync subsecond. However, as a protocol, NTP only guarantees 1 second accuracy.
Is that a hard bound or an average? If it's an average, then what are the limits of the bounds, both in magnitude and duration?
> and PTP can get down to 1e-9 seconds
We use it for audio, and the reason it works well there is because there is no exponential backoff with your peers, allowing for even small timing slips to become enough to notice. 1ms of latency is far too much for our application, we typically aim for 0.25ms.. and we're only running 96kHz. If we lose PTP sync, we notice within a few minutes.
Another advantage of PTP is it can operate as a broadcast and, as the article notes, switches can be PTP aware and help update the timing as the broadcast flows through the network. Conveniently, PTP also allows for multiple timing domains and masters to co-exist on the same network.
It's also an absurdly simple protocol, you can build a receiver for it in about 200 lines of C code. I've actually become quite taken with it since it was forced into our space about 10 years ago.
I know that in trading, auditing trades / order books requires extremely accurate timing, and they typically deploy GPS hardware to get the required level of accuracy. As GPS is accurate to a level of 30ns, 1e-6 to 1e-9 (1ns) is exactly the kind of improvement needed to not need GPS hardware anymore.
AIUI, you cannot quiet think of it like a regular database where a particular row has a particular value which would necessitate only one writer doing (atomic) updates at a time.
Rather it is a MVCC-like database and a bit like an append-only log: as many writers as needed can write and there are 'multiple values' for each row. The "actual" value of the row is the one with the highest transaction ID / timestamp. So updates can happening without (atomic) locking by just adding to the value(s) that already exist.
When reading, applications just generally get served the value with the highest-value timestamp, and since time is synchronized to such a tiny interval, it is a reasonably sure bet that the highest value is the most recent transaction.
This is similar in concept to a vector clock (see also Lamport):
* https://en.wikipedia.org/wiki/Vector_clock
But instead of logical clocks with 'imaginary time', 'real time' is used down to the sub-microsecond level.
NTP can work via broadcast and multicast.
You're simply trading NTP hardware for PTP hardware (grandmaster clocks). There is no way to get to 1e-9 scales without hardware support.
1ns of error equates to ~30cm of error (from speed of light), so:
You might get a couple of ns of error from the initial s and p waves. (Assuming on the order of 50cm displacement)
If you’re on the fault and there is a major displacement (say order of 1-2 meters) there might be several more ns of error.
Practically, this seems to suggest that by the time that displacement from an earthquake would cause a significant loss of clock accuracy (say > 10ns), you most likely have bigger problems to worry about..
I assure you, with the right hardware and paying attention to your latencies, NTP can get you down below one millisecond accuracy. Poul Henning Kamp was doing nanosecond level accuracy with NTP back in the mid-aughts, but then he had rewritten the NTP server code, the NTP client code, and the kernel on the server.
As an NTP service provider, what you really want to keep an eye on is the Clock Error Bound that gives you the worst case estimate for how bad the time is that you could be serving to your customers. For the client side, you mainly care about just the accuracy you're actually getting.
PTP also requires some intelligence with regards to configuring your Ordinary Clocks, your Transparent Clocks, and your Border Clocks. And you have to have these configured on every device in the network path.
PTP does have a unicast mode as well as multicast, which can help eliminate unknowable one-way latencies.
It's a pain.
Check the documentation at https://linuxptp.nwtime.org/documentation/ and especially https://linuxptp.nwtime.org/documentation/ptp4l/
I already have an ancient Meinberg Stratum-1 somewhere that I should pull out of storage and send back to Heiko so that they can put it in a museum. These days, for proper datacenter use, I'd go for something like this one: https://www.meinbergglobal.com/english/products/modular-2u-s...
Feel free to look at the documentation I linked above.
But NTP is something that many can deal with.
So, use PTP as a refclock for your NTP service, and then let the clients deal with an interface they already know and understand.
Those clients that really care can deal with the additional complexities of PTP, if you want to let them.
We've got more cool stuff we are working on.
If we can get past this hiring freeze problem, we still have a couple of slots we need to fill. If you know a lot about time sync and the NTP and PTP protocols, and you have a passion for building some of the biggest impact systems on the planet, let me know.
You wouldn't want to build your entire monitoring system on top of that and be forced to deploy those at hundreds of datacenters around the world.
The "one second" number is not inherent to the protocol, but comes from a survey from 1999:
> A recent survey[2] suggests that 90% of the NTP servers have network delays below 100ms, and about 99% are synchronized within one second to the synchronization peer.
* http://www.ntp.org/ntpfaq/NTP-s-algo.htm#Q-ACCURATE-CLOCK
A 2005 survey found (AFAICT, see Figure 1) that north of 99% — more like 99.5% — of servers had offsets less than 100ms, and that 90% have offsets less than 10ms:
* PDF: https://web.archive.org/web/20081221080840/http://www.ntpsur...
The output of "chronyc tracking" from a randomly-selected system I help run:
Reference ID : […]
Stratum : 3
Ref time (UTC) : Wed Nov 23 13:35:21 2022
System time : 0.000002993 seconds fast of NTP time
Last offset : +0.000003275 seconds
RMS offset : 0.000008091 seconds
Frequency : 13.191 ppm slow
Residual freq : +0.001 ppm
Skew : 0.017 ppm
Root delay : 0.001615164 seconds
Root dispersion : 0.000048552 seconds
Update interval : 65.3 seconds
Leap status : Normal
All I had to do to achieve this was have some infra nodes (that do other things as well) point to *.pool.ntp.org, and then have my cattle and pets point their NTP software to those infra nodes. No special hardware, no special software, no configuring of network switches and routers: just a few lines of Ansible and I get (estimated) microsecond (1e-6) error levels.
In general, once you need/want anything past what NTP can deliver for you, you really, really need to understand your use case and then design your time services to meet your specific needs.