John Carmack's arguments against building a custom XR OS at Meta

Intel still does it. As far as I can see they're the only player in town that provide open, detailed documentation for their high-speed NICs [0]. You can actually write a driver for their 100Gb cards from scratch using their datasheet. Most other vendors would either (1) ignore you, (2) make you sign an NDA or (3) refer you to their poorly documented Linux/BSD driver.

Not sure what the situation is for other hardware like NVMe SSDs.

[0] 2750 page datasheet for the e810 Ethernet controller https://www.intel.com/content/www/us/en/content-details/6138...

On the other hand, see the complete mess that are the IPU6/7 camera chipsets and their Linux support.

The NVMe spec is freely downloadable and sufficient to write a driver with, if your OS already has PCIe support (which doesn't have open specifications). You don't need any vendor-specific features for ordinary everyday use, so it's a bit of a different situation from NICs. (Also, NVMe was in very large part an Intel creation, though it's maintained by an industry consortium.)

Wow... that PDF is 2,750 pages! There must be an army of technical writers behind it. That is an incredible technical achievement.

Real question: Why do you think Intel does this? Does it guarantee a very strong foothold into data center NICs? I am sure competitors would argue two different angles: (1) this PDF shares too much info; some should be hidden behind an NDA, (2) it's too hard to write (and maintain) this PDF.

In terms of (2), I wonder if it's even possible to write a driver without such a document. In the end, the vendor is on the foot for the driver for major platforms (let's assume Linux) - if they can write a Linux driver without a similar spec to this doc, then the doc probably doesn't need to exist since the business wins from hobbyist drivers will be low. If they can't though, then it's just a matter of formatting an internal document for public consumption - the doc itself has to be maintained anyways so the cost seems lower and maybe reasonable. I have a feeling the doc is necessary but I am not specialized in the field.

Assumptions, fair or not, about (1) seems more likely somehow.

Id wager high frequency trading applications.

This is a pretty standard document length. Modern microcontrollers have similar lengths (e.g. ATSAMD51 is ~2000 pages). Some of it is not software related, things like pin outs and electrical and mechanical descriptions.

It does take a huge amount of work to write and maintain. Typically the authors are not technical, so it also relies on the designers being available to answer questions as well. Then there’s a choice of how it’s written: narrative and potentially imprecise but readable, or terse and precise but hard to read. There’s both styles in the same document, terse for register descriptions.

Didn't all the asahi Linux Mac m1 drivers essentially get reverse engineered with little to no support from apple and no public docs? If I'm remembering correctly then I guess it's possible with enough effort and reverse engineering skills

Good christ this is my current work laptop. It...mostly doesn't work. Plug in a USB camera and it'll just go. Several drivers, userspace utilities and other daemons and sometimes gstreamer works, but does Zoom work? Who knows!

But it took 5 years. And since the first model, there are many others. It is huge work.

For datasheets that's normal. Might even be leaning towards smaller than average for the device in question.

For comparison, a data sheet for a single transistor can be around 12 to 30 pages. A data sheet for a tiny microcontroller is probably a few hundred pages.

I once wrote a driver for a flash chip and that had a data sheet of around 80 pages.

Probably CPU vendor culture? I forgot how large Intel's manual set is, but ARM's was ~11k pages the last time I checked. Intel's was smaller, but not that much smaller, certainly within an order of magnitude.

I may be the only person who ever understood every detail of C++, starting with the preprocessor. I can make that claim because I'm the only person who ever implemented all of it. (You cannot really know a language until you've implemented it.) I gave up on that in the 2000's. Modern C++ is simply terrifying in its complexity.

(I'm not including the C++ Standard Library, as I didn't implement it.)

P.S. we're adding an "Editions" feature to D so we can simplify the language by removing obsolete and deadend features. We didn't get everything right, and want to fix it!

It was reverse engineered from a driver. With no driver and purely some PCIE device registers mapped into memory you might as well be trying to guess lottery numbers.

This is one thing Rust did it right and I hope more languages adopt this.

I guess the driver was the one that runs on Mac that they were able to refer to? Not sure you have any links to blog posts about this process it sounds so cool

Look up the Texas Instruments am3358. It's a tiny SOC, it was used in the beaglebone black. Its technical reference manual[1] is over 5000 pages, and it details all peripherals, all of the interconnects and every single register in the system. This, by contrast, is really just an overview.

Regards to (1), if you don't publish this information you're not selling a CPU, you're selling a very expensive chunk of sand. There is simply no way that a customer can guess at what your implementation looks like. Additionally, Intel barely has IP in the traditional sense. They hold patents, but their only real competitor in making x86 processors, AMD, has a long-standing mutual non-enforcement agreement wrt patents.

Regards to (2), I'm guessing a majority of this PDF can be generated sort of like you generate API documentation from doxygen comments.

[1]: https://www.ti.com/lit/ug/spruh73q/spruh73q.pdf?ts=175651560...

From https://news.ycombinator.com/item?id=39385382 Here's Hector Martin (marcan) talking about their tooling: https://asahilinux.org/2021/08/progress-report-august-2021/

Running macOS in a hypervisor controlled from a second machine (proxy mode + verbose==on) to watch drivers talk to the hardware:

https://asahilinux.org/docs/sw/m1n1-user-guide/#running-a-ma...

You've done some great work but I have to call BS with this claim:

> I'm the only person who ever implemented all of it.

Sean Baxter is an easy counter example.

I don't know much of anything about him. Did he implement the preprocessor? the optimizer? the code generator?

(For some context, back in the 80's, code generators needed enhancements to implement C++. You couldn't just use an existing one. Bjarne had to do some ugly workarounds because of this.)

That's interesting that it's that short. I remember a long while ago I had aspirations of implementing a custom board for Prestonia-/Gallatin-era Xeons and the datasheets and specs for those was around 3000 pages, iirc. Supporting infra was about that long as well. So I'm surprised to see a modern ethernet controller fit into the same space. I appreciated all of the docs because it was so open, I felt like I could actually achieve that project, but other things took priority.

> Real question: Why do you think Intel does this?

I'm not sure large traditional silicon vendors like Intel, TI, et al re-evaluate the documentation requirements (and costs) on a chip by chip basis. It's probably done by chip class and for companies who've been selling chips by the millions over many decades to industries as diverse as defense, aerospace, automotive, etc there are classes of chips where robust, complete documentation is not only expected but often a required part of the RFP, compliance or conformance processes.

While this level of effort probably isn't needed for every chip in that class, it could be hard to reliably predict when a general purpose chip is still in the design phase which customers may be interested in it during its life (which for some of these chips might be decades). Many chips which conform to MIL-SPEC or other similar standards which can require extensive documentation are simply enhanced versions of standard chips, so the docs exist anyway. Finally, there's the organizational capabilities and culture aspect. Once the org needs to maintain the systemic ability to generate serious documentation at scale, you end up with a lot of managers and staff who think this way.

Sean Baxter single-handedly implemented all of up to C++23, and some C++26, including a huge number of GNU extensions and possibly an even larger number of his own features.

I worked on a similar TI SoC -- with War-and-Peace-sized datasheet. My eyes burned out and brain exploded. Ultimately, another engineer had to take over the project -- or rather TEAM of engineers, of which I did only a part. It's simply to much complexity to expect one engineer to grok it all, do the schematic & PCB & power supply & hi-speed MIPI connections and radios and... and THEN to write the software for it all. It's too much. (This is the Life one gets in Startups, it seems -- worked to the (beagle)bone!)

Sean Baxter's circle compiler uses LLVM as a backend, but I believe the rest is from scratch.

Arguably these days having a clear frontend/backend separation is good compiler architecture. It might slow down compile times a bit, but it's worth the cost.

It wouldn't have made much sense to write the preprocessor these days, too, but it is part of the C++ compiler. Unless integrating it with the C++ lexer for speed purposes, as I did.

Impressive!

So it sounds like he wrote the frontend of a cpp compiler? There's a lot of work in other layers as well.

having used the AM3358 extensively, the TRM is not complete. There are some pretty important and complex systems that have literally no documentation at all in the TRM, not to mention the large number of quirks and small details that you can only pick up from a scattering of other areas (including a wiki that TI deleted some years ago). It is, however, miles better than the documentation available for most SOCs.

Thanks for taking the time to round up these posts for me!

Now, who can claim such bold... oh, it's YOU.

Agreed. The actual technical writing portion of the TRM leaves much to be desired.