Writing an operating system kernel from scratch

I will never not find this kind of project incredibly impressive. It’s interesting to think that Linux, after all, is really just the kernel — and yet getting that work done paved the way to getting an open source version of Unix installed on billions of machines. Great stuff!

It is equally valid to say that Stallman's starting to write a C compiler and Unix utilities (in 1984 whereas the Linux project started in late 1991) paved the way to getting an open source version of Unix installed on billions of machines.

Zig really is amenable to OS development. And so is RISC-V. I started this same exercise except with x86 first. I quickly got annoyed with all the legacy boilerplate required. RISC-v doesn’t have any of that. It’s so much faster to get up and running. Here’s my bad zig: https://github.com/Fingel/aeros-v

How does one get their hands on low-cost RISC hardware?

> with x86 first. I quickly got annoyed with all the legacy boilerplate required.

IMHO, if you use a reasonable bootloader, you don't have too much boilerplate. Multiboot loaders do leave you in real mode, and most people want to be in protected mode, so you have to set up some tables and do a jump, but that's not that much boilerplate. There's a bit more stuff if you disable legacy interrupt controllers (which you probably want to) but it seems to me being able to maybe run on a regular pc is worth it (caveats about console interfaces apply... my hobby OS needs bios boot and uses some vga things that I found aren't well supported... serial console is easier, but lots of computers don't have a serial port either)

There's this $10 board on Aliexpress called Milk-V Duo S. It's been popping up on my recs every now and then. Looks interesting.

https://aliexpress.com/w/wholesale-Milk%2525252dV-Duo-S.html

Taken directly from its description:

>Milk-V Duo S is an upgraded model of Duo with an upgraded SG2000 master with a larger 512MB RAM and extended IO capability

>Onboard WI-FI6/BT5(Model Milk-V-Duo-S-512M-Basic/Milk-V-Duo-S-512M-eMMC does not have this function)

>USB 2.0 HOST port

>100 Mbps Ethernet Portwith PoE Support (via PoE HAT)

>Dual MIPI CSI with ISP

>The device also supports switch between RISC-V and ARM boot via a switch

I agree - there's a number of kernels that were "open source" and released at a similar time enough time to linux (e.g. 386BSD in '92) that I could see any of those winning the "community battle" and taking that space instead, but no real credible "development toolchain" equivalent until decades later.

Though I'm unsure how differing licenses might have affected this - I suspect that really early in it's development the "copyleft" nature of the GPL Linux didn't make as much of a difference, as from what I remember most commercial uses of Linux didn't come until it had already gained significant momentum.

The copyleft nature was essential to good driver support. It set it up such that for corporations making drivers the easiest path was to get the driver upstreamed. There was a bunch of hoops they could have gone through to avoid that (as many did, like Nvidia) but that became a sorta-default.

Copyleft encourages a collaborative relationship between entities because it makes trying to play it close to the chest with IP involve more legal effort (if it's possible at all).

It's even more funny and amusing when you remember that in the initial release mail for Linux, Torvals said "just a hobby, won't be big and professional like gnu".

https://groups.google.com/g/comp.os.minix/c/dlNtH7RRrGA/m/Sw...

I think there are dozens of boards by now, but what I found interesting and pledged to is this:

https://www.kickstarter.com/projects/starfive/visionfive-2-l...

I don't have a first version visionfive2 myself, but i heard good things about it and the eco-system sounds like it's growing. There are still rough edges. Lets hope they ship soon (tm)

Edit: What I currently have for my personal projects is a PolarFire SoC Discovery Kit. That's a quad-core RISC-V system with FPGA embedded. Maybe too expensive and not for everyone (130 bucks, fun fact: The dev board is cheaper than the chip itself).

https://www.microchip.com/en-us/development-tool/MPFS-DISCO-...

Also the microchip documentation, toolchain and so forth feel really crappy and old, but once you get used to it, it's not that bad and actually getting to run bare-metal risc-v code is easy. There are easy examples for linux and bare-metal

> I will never not find this kind of project incredibly impressive

I wouldn’t call it incredibly impressive. The path on how to write a minimal multi-tasking kernel has been beaten decades ago.

Writing a kernel that can boot and do a few things is ‘just’ a matter of being somewhat smart and have some perseverance. Doing it for RISC-V complicates things a bit compared to x86, but there, too, the information about initialising the hardware often is easily obtained (for example: https://wiki.osdev.org/RISC-V_Meaty_Skeleton_with_QEMU_virt_... I wouldn’t know whether this project used that)

I think the author agrees, given (FTA) that they wrote “This is a redo of an exercise I did for my undergraduate course in operating systems”

It’s work, may be nice work, but I think everybody with a degree in software engineering could do this. Yes, the OS likely will have bugs, certainly will have rough edges, but getting something that can multi-process, with processes shielded from each other by a MMU isn’t hard anymore.

Yes, I can see that stalling development as (at best) it turns into a pile of private forks rather than a cohesive project, but from what I remember that was already after Linux had "won" the "Open Source Kernel" race.

Commercial support for Linux was... Sparse... before the early 2000s.

He's also referred to Linux as "the GNU Emacs of all terminal emulators".

> I think everybody with a degree in software engineering could do this

Ideally this would be true, but it hasn't been my experience at all. At least with American graduates, I can't speak to other countries.

> El acceso a la presente dirección IP ha sido bloqueado en cumplimiento de lo dispuesto en la Sentencia de 18 de diciembre de 2024, dictada por el Juzgado de lo Mercantil nº 6 de Barcelona en el marco del procedimiento ordinario (Materia mercantil art. 249.1.4)-1005/2024-H instado por la Liga Nacional de Fútbol Profesional y por Telefónica Audiovisual Digital, S.L.U. > https://www.laliga.com/noticias/nota-informativa-en-relacion...

Of FFS...

one thing is using http but another this one…

...what? a football organization in Spain has the power to restrict internet access to the entire country?

Is there a book one can read to learn how to create one?

You still have to set up the GDT, TSS, all that other legacy stuff that isn't actually used but it's still there. And if you set it up incorrectly, your punished for it with triple faults and the like, even if nobody even uses anything like the GDT or TSS these days.

Submitted separately by the author:

https://news.ycombinator.com/item?id=45236479

Copying their comment to here:

I have redone the classical exercise of writing a tiny OS kernel with time sharing, which manages a couple of user threads. My goal was to experiment specifically on RISC-V + OpenSBI. Additionally, I wanted to explore Zig a little bit, so that was the language used instead of the traditional C, but it should be straightforward how to do the same experiment in either C or Rust.

It's definitely very rough around the edges, and it's more of an experiment and an intro for people who want to go through step 0 of learning OS kernel development and computer architecture. Nevertheless, I hope it is still a fun experimental thing to play with over the weekend!

The full walkthrough and the GitHub link are available at the link posted!

Football is really big in Spain dontcha know?

To you maybe. The subset of population that is even interested, smart, persevering to do this is extremely tiny.

I heard about this situation in Spain where Cloudflare (and maybe others) are blocked by mistake due to being associated with some sites streaming football.

How are people getting around this problem, I guess VPN? I mean, some of the wrongly blocked IPs are needed for work I imagine.

I’d like to have a go at this. What are you using to run your risc-v kernel? Just Qemu or can you recommend any real hardware?

Just Qemu. I used this as a guide: https://operating-system-in-1000-lines.vercel.app/en/ It's C so translating to Zig is pretty fun.

> [Stallman/GNU] getting an open source version of Unix installed on billions of machines.

Agreed, funnily enough GNU tools/compilers also ended up getting installed on a lot of proprietary UNIXes because proprietary UNIX was mostly shit (in user space!). At least most of the ones I had the misfortune to have to work on.

Yes, that is exactly what is happening

> It's now the GNU Emacs of all terminal emulators.

> (Linus Torvalds, regarding the fact that Linux started off as a terminal emulator.)

http://neil.franklin.ch/Jokes_and_Fun/Linux_Quotes.html

That's the best reference I can find, but even if it's totally legit it doesn't make any sense to me.

that is a pain. oswiki is a good resource, and there are also a huge number of examples. but it still takes a day. context save and restore and interrupts is another speed bump. but its no different than any other development banging your head against the wall exercise, and its pretty rewarding to have smoothed all that over and go to town on bespoke scheduler design.

Perhaps it's a reference to how Emacs does "everything" and some has also joked that Emacs is an OS with a text editor

My CS undergrad in the UK had us write an ARM kernel with scheduling and IPC, though didn’t require we use the MMU.

I first came across GNU tools on NeXTSTEP, which wasn't too bad.

If Stallman had started with a kernel, there would be very few people who had the legal right to run any utilities or apps on the new kernel whereas GNU's utilities and apps (e.g., Emacs) were immediately useful (i.e., without breaking any copyright law or violating any software license) to a large population, namely, anyone with an account on a proprietary Unix system, which explains why Stallman chose to start with the userland.

For those who want a "packaged" experience doing this same kind of work, there's "Operating System in 1000 Lines of Code" (https://operating-system-in-1000-lines.vercel.app/en/), I followed it a while ago in Zig (which meant translating C code snippets to Zig) and had a good time. I have the code and VODs here: https://github.com/kristoff-it/kristos/

I dunno, I've done it myself, and I would say that it's qualitatively different than other "banging your head against the wall" software development. At least it's an order of magnitude more painful, due to lack of integrated tooling, questionable documentation, and a more delayed feedback loop.

Single-threaded kernel bringup is easier than most distributed systems though.

Always a fun and interesting thing to try.

Also go ahead and try hard things like your own cryptography. The advice “don’t roll your own crypto” just means don’t DEPLOY something that is not battle tested, because it is very easy to make BIG mistakes in that area. Always feel free to experiment and explore.

We need more operating systems, more choice.

That is an excellent recommendation. For Operating Systems anything Andy Tanenbaum did is world class.

This made me look up what he has been up to, there is a 2023 edition of "Modern Operating Systems" which covers cloud virtualization and Android along with everything that came before, hm, tempting.

I'd suggest https://os.phil-opp.com/ instead of a book from long ago. There is no reason to drag yourself through C anymore.

And if you do want the more historical content, https://www.projectoberon.net/

nice!

i love how accessible riscv isa is, the docs are amazing, there are tons of examples, and tons of emulators, the uncompressed machine code is quite readable.

i am writing a book for my daughter and am also working on a small os with forth and timesharing https://punkx.org/projekt0/book/part1/os.html and i dont think i could've done it on x86, learning both forth and riscv assembly on the way, its super fun.

honestly if you ever wanted to make a toy os from scratch there is no better time to start than now (besides maybe 1980), get a cheap riscv like rp2350 and just upload the relevant sections of the docs to claude and it will help you if you get stuck.

Maybe just run an emulator on your x86/apple, for now? It'll probably be faster than actual hardware, and it will for sure be faster to develop a kernel with.

https://www.qemu.org/docs/master/system/target-riscv.html

Funnily enough, when I was writing one in Zig for x86 I actually was the one who found a bug in (I think) 0.11/0.12 where packed structs broke when you assigned to their fields (and I also found that the `.interrupt` calling convention was a todo item in that version). GDT/TSS is always, always one of the most annoying parts of an x86 OS for me, because all of it is so tedious and unnecessary, and I wish Intel would just get rid of IA32 mode already, or add some kind of bit in CR0 or CR4 which caused all that extraneous setup of legacy structures to just cease to be. RISC-V is so, so much more pleasant to get started with. Limine helps regardless of which architecture your getting going with; it's pretty much my de facto bootloader at this point.

Setting up the GDT is exactly the point where I got annoyed enough to see what risc-v was all about.

I am 99.99% sure that less than 20% of Australian graduates could do this, and honestly I wouldn't be surprised to hear that the actual answer is <1%.

I was studying at Monash, which is considered a solid university here, and holy moly are the standards low. I had classmates in the second year of my machine learning postgrad asking me things like "What is machine learning?", and they all graduated and found jobs anyway.

We really need a Tech Writer Hall of Fame. W. Richard Stevens, Andrew Tanenbaum, P.J. Plauger. Others?

What did they find jobs in? Australia has like one tech company.

Which is odd since their universities have built two of the most interesting CS projects I can think of (Mercury and L4). And WWWJDIC I suppose.

Yep, agreed. Don't get me wrong, the Intel/AMD docs are amazing. But even getting the hardware to a usable state (where I define usable as "something that can enumerate PCI devices on the system, even without PCIe") is an absolute pain that really just shouldn't exist. I know that RISC-V ahs it's own tough spots but... IMO it's a lot faster to get up and running with because it doesn't have 10000 execution modes with tons of different requirements you need to do for each mode, and on and on and on and on. (Yes, I'm exaggerating a bit.)

I'd agree with this. I did a double degree in Comp Sci/Comp Sys Eng at RMIT (1998-2002) and even from that era I would say that's largely true. Out of the people who did my course (and those I knew from other degrees like Comp Sys Eng/Business) very few are still doing deep technical programming for a career and/or hobby programming on the side on deep technical non-web things. The rest are mostly working for places like consulting companies, banks, big data, Telstra, etc in management roles like project manager, scrum master, solutions architect, change management. A lot of folks I think were just not that interested in stuff like writing an OS, how does virtual memory work, how does the hardware work, etc so they gravitated out of those software development roles into management roles. Nothing wrong with that, but I just think not everyone is interested in or capable of writing an OS!

Folks end up at all sorts of places. Like I mentioned above, the banks hoover up a lot of graduates. There are a lot of smaller local companies doing web stuff. The consulting companies all have a presence here (KPMG, Accenture, Fujitsu Consulting, etc).

Kernighan!

And how can we leave out the OG of tech writers: Donald Knuth. He got a bit distracted by developing TeX but he got a well deserved Turing award for the series.

Naturally, it is basically revisiting the safety of Object Pascal and Modula-2, and repacking it into a C syntax.

C was never special, beyond being widely available thanks to UNIX original licensing.

I just checked the prices for my country and I’m surprised how cheap this is. I’m 90% sure I’m buying one just to play around with.

This has risc-v: https://www.raspberrypi.com/products/raspberry-pi-pico-2/

and the datasheet is amazing https://datasheets.raspberrypi.com/rp2350/rp2350-datasheet.p...

there is an issue that language models have seen enormous amounts of rp2040 code and docs, and it they get properly confused a lot, so you should just pdf print relevant pages and use them in-context.

    > If Stallman had started with a kernel, there would be very few people who had the legal right to run any utilities or apps on the new kernel

That is really not true, one of the most important things when it comes to the GNU project and the whole Free Software movement is the ability to run _any_ program, be it non-free software or free software. This has been parroted for more than 40 years now ...

What would be the level of Zig knowledge needed to do this exercise?

Or, asked differently, with no knowledge of Zig, what would be a realistic approach to trying this out? (Assuming an interest in learning Zig, with a background in C++)