lsr: ls with io_uring

Love it.

I'm trying to understand why all command line tools don't use io_uring.

As an example, all my nvme's on usb 3.2 gen 2 only reach 740MB/s peak.

If I use tools with aio or io_uring I get 1005MB/s.

I know I may not be copying many files simultaneously every time, but the queue length strategies and the fewer locks also help I guess.

One reason is so that they work in all linux environments rather than just bleeding edge installs from the last couple years.

Probably historical preference for portability without a bunch of #ifdef means platform+version-specific stuff is very late to get adopted. Though, at this point, the benefit of portability across various posixy platforms is much lower.

Poe's law hits again.

Has anyone written an io_uring "polyfill" library with fallback to standard posix-y IO? It could presumably be done via background worker threads - at a perf cost.

Seems like a huge lift since io_uring is an ever growing set of interfaces that is encompassing more and more of the kernel surface area. Also, the problem tends to not necessarily be that the io_uring interface isn’t available at compile time but a) the version you distribute to has a kernel with it disabled or you don’t have permission to use it meaning you need to do LD_preload magic or use a framework b) the kernel you’re using supports some of the interfaces you’re trying to use but not all. Not sure how you solve that one without using a framework.

But I agree. It would be cool if it was transparent, but this is actually what a bunch of io-uring runtimes do, using epoll as a fallback (eg in Rust monoio)

iirc io_uring also had some pretty significant security issues early on (a couple of years ago). Those should be fixed by now, but that probably dampened adoption as well.

Thats a great speed boost. What tools are these?

io_uring is a security nightmare.

How so?

This is a good read on the topic: https://chomp.ie/Blog+Posts/Put+an+io_uring+on+it+-+Exploiti...

you give process direct access to a piece of kernel memory. its a reason why there is separation. thats all.

Most of the security concerns with io_uring that I've seen aren't related to the shared buffers at all but simply stem from the fact that io_uring is a mechanism to instruct the kernel to do stuff without making system calls, so security measures that focus on what system calls a process is allowed to do are ineffective.

This isn't the issue; it's relatively easy to safely share some ring buffers. The issue was/is that io_uring is rapidly growing the equivalent of ~all historical Linux syscall interfaces and sometimes comparable security measures were missed on the new interfaces. (Also, stuff like seccomp filters on syscalls are kind of meaningless for io_uring.)

io_uring is the asynchronous interface and that requires to use even-based architecture to use it effectively. But many command-line tools are still written is a straightforward sequential style. If C would have async or similar mechanism to pretend doing async programming sequentially, it would be easier to port. But without that a very significant refactoring is necessary.

Besides, io_uring is not yet stable and who knows may be in 10 years it will be replaced by yet another mechanism to take advantage of even newer hardware. So simply waiting for io_uring prove it is here to stay is very viable strategy. Besides in 10 years we may have tools/AI that will do the rewrite automatically...

...don't you supply the memory in the submission queue? or do you mean the queues themselves?

Not years ago. io_uring has been a continuous parade of security problems, including a high severity one that wasn't fixed until a few months ago. Many large organizations have patched it out of their kernels on safety basis, which is one of the reasons it suffers from poor adoption.

> If C would have async or similar mechanism to pretend doing async programming sequentially, it would be easier to port.

The *context() family of formerly-POSIX functions (clownishly deprecated as “use pthreads instead”) is essentially a full implementation of stackful coroutines. Even the arguable design botch of them preserving the signal mask (the reason why they aren’t the go-to option even on Linux) is theoretically fixable on the libc level without system calls, it’s just a lot of work and very few can be bothered to do signals well.

As far as stackless coroutines, there’s a wide variety of libraries used in embedded systems and such (see the recent discussion[1] for some links), which are by necessity awkward enough that I don’t see any of them becoming broadly accepted. There were also a number of language extensions, among which I’d single out AC[2] (from the Barrelfish project) and CPC[3]. I’d love for, say, CPC to catch on, but it’s been over a decade now.

[1] https://news.ycombinator.com/item?id=44546640

[2] https://users.soe.ucsc.edu/~abadi/Papers/acasync.pdf

[3] https://www.irif.fr/~jch/research/cpc-2012.pdf

> I'm trying to understand why all command line tools don't use io_uring.

Because it's fairly new. The coreutils package which contains the ls command (and the three earlier packages which were merged to create it) is decades old; io_uring appeared much later. It will take time for the "shared ring buffer" style of system call to win over traditional synchronous system calls.

Last I checked it's blocked by most container runtimes exactly because of the security problems, and Google blocked io_uring across all their services. I've not checked recently if that's still the case, but https://security.googleblog.com/2023/06/learnings-from-kctf-... has some background.

https://security.googleblog.com/2023/06/learnings-from-kctf-... - Has some interesting information on that topic.

I updated the Wikipedia article on io_uring to dispute that.

You can just ask io_uring what commands you have available to you. Though the way of the background thread should be readily available/usable by just indirectly calling the syscall (-helper) and replacing it with a futex-based handshake/wrapper. If you're not using the backend-ordering-imposing link bit, you could probably even use minor futex trickery to dispatch multiple background threads to snatch up from the submission queue in a "grab one at a time" fashion.

Iouring is very recent

The memory for the submission queue is mmapd into user space. Easiest implementation to read is the Zig stdlib:

https://github.com/ziglang/zig/blob/69cf40da600224734d39c6f6...