John Carmack's arguments against building a custom XR OS at Meta

These days, you get a medium-level description and a Linux driver of questionable quality. Part of this is just laziness, but mostly this is a function of complexity. Modern hardware is just so complicated it would take a long time to completely document, and even longer to write a driver for.

The people who develop OSes are cut from a different cloth and are not under the usual economic pressures.

(And I feel bad saying this since Meta obviously did waste eleventy billion on their ridiculous Second Life recreation project ...)

That's what's claimed. That's what people say, yet it's just an excuse. I've heard the same sort of excuse people have, after they write a massive codebase, then say "Oops, sorry, didn't get around to documenting it".

And no, hardware is not more difficult than software to document.

If the system is complex, there's more need to document, just as with a huge codebase. On their end, they have new employees to train up, and they have to manage testing. So any excuse that silicon vendors have to deal with such immense complexity? My violin plays for them.

So please don't mock the spend. Big spends fail sometimes, and at least people were paid to do the work.

If they'd spent the money researching nuclear fusion or space flight or a new way to develop microprocessors, I would be cheering their efforts even if they had failed in the end.

On the other hand, Meta's experiment is primarily CEO-driven. The outcome is predetermined, changing direction is not possible. Sure, clever engineers get to draw the rest of the owl, but that's not very useful when it turns out that everyone needs a horse instead.

They are spending a fortune, but rather than getting 900 crappy ideas to throw away and 100 great ones to pick from for continued development, they are developing 1 technological marvel nobody is interested in.

It’s not first party documentation that’s the problem. The problem is that they don’t share that documentation, so in order to get documentation for an “unsupported” OS a 3rd party needs to reverse engineer it.

What is an easy gate task to get into “reverse engineering some drivers for some OS”?

Second thought: I don’t even know how to write a driver or a kernel, so I better start from there.

Not the parent, but of course they're wasting their time... That's the point of a hobby OS.

I'm working on a hobby OS, and I have no illusions that it's most likely fewer than 10 people will ever run it, and less than 100 will hear about it, but it lets me explore some interesting (to me) ideas, and forces me to learn a little more about random pieces of computing. If I ran on GCP, I'd want the reboot button to work. That sounds useful.

On the topic, I don't see why anyone would want to build a general purpose OS. There's enough already and even with the shrinking of hardware variety, there's a lot of stuff to support to make a general purpose OS work on enough hardware for people to consider using it. You can take Linux or a BSD and hack it up pretty good to explore a lot of OS ideas. Chances are you're going to borrow some of their drivers anyway, and then you'll end up with at least some similarity... may as well start there and save a lot of time. (My hobby OS has a custom kernel and custom drivers, but I only support a bare minimum of devices... (pc) console i/o, one real NIC, and virtio-net... that's all I need; I might add support for more NICs and more consoles later)

They aren't trying to get reboot to work, they are trying to get their version of kexec to work so their hobby os reboots faster.

https://wiki.archlinux.org/title/Kexec

The biggest scam in the OS world is drivers, we should demand more out of our hardware. Drivers shouldn't be necessary.

That's obviously the wrong message. They should say "Go ask the engineering VP to get us off any other projects for another cycle while we're writing 'satisfying' documentation".

Extensive documentation comes at a price few companies are willing to pay (and that's not just a matter of resources. Look at Apple's documentation)

Not sure what the situation is for other hardware like NVMe SSDs.

[0] 2750 page datasheet for the e810 Ethernet controller https://www.intel.com/content/www/us/en/content-details/6138...

I don't see why a kexec alike wouldn't work about the same on GCP vs qemu vs bare metal... Or what that has to do with a GCP soft reboot button (which again, I think is referring to the reboot button in the GCP console)

Either way, the whole thing is a waste of time, yes? Why not waste time on the part that's engaging?

> The biggest scam in the OS world is drivers, we should demand more out of our hardware. Drivers shouldn't be necessary.

I can't even fathom what you mean here? You've got to have some interface to communicate with hardware. That's a driver. Some hardware only needs a very small driver... Tell the hardware where to send input, how to notify when input is ready and when its ready for output, and tell the hardware where data to output is. Maybe some setup stuff for modes and whatever if the needs aren't obvious and universal. I don't see how you could possibly avoid that.

It would certainly be possible for more devices to use common interfaces so a single driver could operate many different devices. Maybe that's what you mean? There's some movement towards that... SATA controllers generally speak AHCI, human interface devices generally appear as USB HID devices, etc. NICs tend to have a wide variety of setup sequences, but data queues usually fit into one of a limited number of patterns.

It helps to have a concept to guide you too, but you can certainly make some progress on the basics before you figure out what you really want to do.

[1] https://wiki.osdev.org/User:Zesterer/Bare_Bones

[2] https://osdev.wiki/wiki/Multiboot1_Bare_Bones

Realistically for the OEM to debug the issue they're going to need a way to reliably repro which we don't have for them, so we're kind of stuck.

This type of problem generalizes to the development of drivers and firmware for many complex pieces of modern hardware.

This way, good documentation is priced into my estimate for the project. I don't have a work item "spend a few days documenting." Nope, if I'm doing a foo then that includes documenting a foo at the same time.

Of course, your custom kernel will still have to have some of its own code to support core platform/chipset devices, but LKL should pretty much cover just about all I/O devices (and you also get stuff like disk filesystems and a network stack along with the device drivers).

Also, it probably wouldn't work so well for typical monolithic kernels, but it should work decently on something that has user-mode driver support.

I agree you need a driver but for most hardware, that should be pretty simple, and easily documented by the hardware vendor, shouldn't it? A button has to be about the simplest possible I/O device imaginable.

thus calling into question why you ever bothered writing a new kernel in the first place if you were just going to piggyback Linux's device drivers onto some userspace wrapper thingy.

Im not necessarily indoctrinated to the point where I can't conceive of Linux being suboptimal in a way which is so fundamental that it requires no less than a completely new OS from scratch but you're never going to get there off of recycling linux's device drivers because that forces you to design your new OS as a linux clone in which cade you definitely did not need to write an entire new kernel from scratch.

Of course, ACPI is supposed to make interfacing with lots of similar things easier, kind of, so there you go.

  > you're never going to get there off of recycling linux's device drivers because that forces you to design your new OS as a linux clone in which cade you definitely did not need to write an entire new kernel from scratch.

probably it would have to be very general in design, but something along the lines of driverkit or iokit might work?

Yes, you can produce a small amount of code faster if you don’t “waste” your time on documentation, but that becomes counterproductive as soon as you can no longer keep the entire codebase in your head.

However it goes into the same direction of the previous commenter, device drivers are intertwined with the OS semantics, even on microkernels, so eventually it ends being just something like POSIX.

What do you mean by that?

Real question: Why do you think Intel does this? Does it guarantee a very strong foothold into data center NICs? I am sure competitors would argue two different angles: (1) this PDF shares too much info; some should be hidden behind an NDA, (2) it's too hard to write (and maintain) this PDF.

What you describe is probably necessary for getting _fast_ Linux compatibility. However, if you are willing to take the overhead of a few layers of indirection, you can probably sandbox the Linux land somewhere, and not have it impact the rest of your design much.

Most hardware access doesn't have to be particularly efficient. And, yes, for the few pieces of hardware that you do want to support efficiently (eg your storage devices or networking, whatever you want to concentrate on in your design) these you can handle natively.

Btw, I would suggest that most people these days should write their toy operating systems to run as a VM on a hypervisor like Xen or similar. The surface to the outside world is smaller that way.

    > Look at Apple's documentation

Just because you spend a lot of your money on R&D, doesn't mean that each R&D project is automatically a good one. You still have to make choices between them.

    > custom Wifi board

Every improvement after that would count as micro-invention in your dichotomy.

PS: Half-joking, you can write some big portions with LLMs but the point stands.

Assumptions, fair or not, about (1) seems more likely somehow.

You know, one'd think that having a complex hardware should make writing a driver easier because the hardware is able to take care of itself just fine, and provide a reasonable interface, as opposed to devices of the yore which you had to babysit, wasting your main CPU's time, and doing silly stuff like sending them two identical initialization commands with 30 to 50 microseconds delay between or whatever.

I guess one exception maybe is Nvidia who have sort of hidden the complexity by moving most driver functionality onto software on the card. At least that's how I understood it. Don't quote me on that.

Why not, though? We used to have e.g. glass teletypes with microprocessors (8080/8051) in them that exposed a serial bus with very neat command protocol that we still use nowadays, that could boot up, init and self-test all on their own.

This is not impossible, but the effort and costs required are substantial and often lose out on a priority basis to just fixing or improving the product itself.

Documenting how to use or interact with it in a specific context, which often includes perspectives on interactions with other components, or e.g. protocols not explicit in the code, deciding where to draw the lines of what can assumed trivial common knowledge and what should be specified or explicitly not specified without notices not to rely on these etc. , that is a different thing.

If it wasn't, then truly as they used to say, the source code would be it's own best documentation (I am a big fan of programming for readability, but even the best readable code, while it will be correct and up to date, will never be enough nor the best for all)

It does take a huge amount of work to write and maintain. Typically the authors are not technical, so it also relies on the designers being available to answer questions as well. Then there’s a choice of how it’s written: narrative and potentially imprecise but readable, or terse and precise but hard to read. There’s both styles in the same document, terse for register descriptions.

For comparison, a data sheet for a single transistor can be around 12 to 30 pages. A data sheet for a tiny microcontroller is probably a few hundred pages.

I once wrote a driver for a flash chip and that had a data sheet of around 80 pages.

It has a rump kernel architecture which makes reusing the drivers almost trivial compared to reusing linus drivers with a new kernel.

This heavily depends on your niche I think. If you're writing closed source vendor software and your client's only guiding light is your documentation, it's 100% true.

If you're working on a 5 people project that evolves at a fast pace, and everyone touching the code is expected to be familiar with the domain and operations, you'll mostly leave comments (todos, meta info, external ticket links etc), not documentation per se.

(I'm not including the C++ Standard Library, as I didn't implement it.)

Without that, we would have probably just switched hw, because the quite obscure bug was in the ASIC, and debugging that on 2005-6-ish hw is just infeasible.

Regards to (1), if you don't publish this information you're not selling a CPU, you're selling a very expensive chunk of sand. There is simply no way that a customer can guess at what your implementation looks like. Additionally, Intel barely has IP in the traditional sense. They hold patents, but their only real competitor in making x86 processors, AMD, has a long-standing mutual non-enforcement agreement wrt patents.

Regards to (2), I'm guessing a majority of this PDF can be generated sort of like you generate API documentation from doxygen comments.

[1]: https://www.ti.com/lit/ug/spruh73q/spruh73q.pdf?ts=175651560...

Getting it to work on GCP meant properly driving something called the Intel PIIX4 controller which was emulated into the VM.

Separately from the OS being able to turn itself off, the OS needs to process a signal received by the hypervisor on this controller to support the hypervisor gracefully shutting it down. Otherwise GCP will wait 90 seconds after it has sent the shut down signal to give up and terminate the VM itself.

The problem I was trying to solve was (a) OS can shut itself down in GCP (b) restarts in GCP from the GCP console would be instant, rather than take 90+ seconds

I was actually just trying to support "power off" in GCP, with the stretch goal of being able to support graceful power off from the GCP console (which is part of supporting power off then power back on restart).

Running macOS in a hypervisor controlled from a second machine (proxy mode + verbose==on) to watch drivers talk to the hardware:

https://asahilinux.org/docs/sw/m1n1-user-guide/#running-a-ma...

I am addressing your comment and eru's question about drivers.

The hardware that would normally need drivers should present itself over a fixed, well documented protocol. Think virtio, or usb device classes but more comprehensive. This would also allow for said hardware to rigorously tested before it ever sees an OS. As it is now, because the hardware is shit and requires a driver, you can't really test the hardware in a way that an OS would expect because it requires the OS driver to even start to function. The job of the OS is now to repair broken hardware.

https://docs.oasis-open.org/virtio/virtio/v1.3/virtio-v1.3.h...

https://wiki.osdev.org/Virtio

https://en.wikipedia.org/wiki/USB_communications_device_clas... (the only good thing to come out of usb)

The trouble with good docs is that they are work to maintain, like good code. If we decide to change this component in a substantial way soon, which we likely will, I'd have to practically rewrite the docs! Why bother?

Because the docs are part of the code. Write the docs.

I don't expect to win this crusade, but I'll keep writing docs anyway. Then later people will modify the code without modifying the docs, and so the docs will be a lie, but still useful, I think.

It's like asking three people who are not closely familiar with a component, but who have worked with it, "what is this thing, how does it work?" You will get three different answers. It would be nice if one of them were a written description straight from the horse's mouth, even if the component is now more of a camel.

> I'm the only person who ever implemented all of it.

Sean Baxter is an easy counter example.

(For some context, back in the 80's, code generators needed enhancements to implement C++. You couldn't just use an existing one. Bjarne had to do some ugly workarounds because of this.)

I'm not sure large traditional silicon vendors like Intel, TI, et al re-evaluate the documentation requirements (and costs) on a chip by chip basis. It's probably done by chip class and for companies who've been selling chips by the millions over many decades to industries as diverse as defense, aerospace, automotive, etc there are classes of chips where robust, complete documentation is not only expected but often a required part of the RFP, compliance or conformance processes.

While this level of effort probably isn't needed for every chip in that class, it could be hard to reliably predict when a general purpose chip is still in the design phase which customers may be interested in it during its life (which for some of these chips might be decades). Many chips which conform to MIL-SPEC or other similar standards which can require extensive documentation are simply enhanced versions of standard chips, so the docs exist anyway. Finally, there's the organizational capabilities and culture aspect. Once the org needs to maintain the systemic ability to generate serious documentation at scale, you end up with a lot of managers and staff who think this way.

Arguably these days having a clear frontend/backend separation is good compiler architecture. It might slow down compile times a bit, but it's worth the cost.

I wish the guy luck on becoming a superstar influencer, since that seems to be his goal, but he can do it without my help.

Then how do devices end up up having drivers for major OSes? It's all guesswork?

It's just that most internal hardware initially only cares only about Windows (or only Linux) so it makes more financial sense to develop a more complex driver than a complex firmware. The equation might change later on but by then you are stuck with the hardware design.