←back to thread

lsr: ls with io_uring

(rockorager.dev)
335 points mpweiher | 4 comments | 18 Jul 25 12:40 UTC | HN request time: 0s | source
Show context
rockorager ◴[18 Jul 25 15:46 UTC] No.44606061[source]▶
Author of the project here! I have a little write up on this here: https://rockorager.dev/log/lsr-ls-but-with-io-uring
replies(6): >>44606274 #>>44606605 #>>44607773 #>>44607936 #>>44608619 #>>44609311 #
tavianator ◴[18 Jul 25 16:27 UTC] No.44606605[source]▶
My bfs project also uses io_uring: https://github.com/tavianator/bfs/blob/main/src/ioq.c

I'm curious how lsr compares to bfs -ls for example. bfs only uses io_uring when multiple threads are enabled, but maybe it's worth using it even for bfs -j1

replies(1): >>44606774 #
rockorager ◴[18 Jul 25 16:40 UTC] No.44606774[source]▶
Oh that's cool. `find` is another tool I thought could benefit from io_uring like `ls`. I think it's definitely worth enabling io_uring for single threaded applications for the batching benefit. The kernel will still spin up a thread pool to get the work done concurrently, but you don't have to manage that in your codebase.
replies(2): >>44607073 #>>44607227 #
mshockwave ◴[18 Jul 25 17:10 UTC] No.44607227[source]▶
and grep / ripgrep. Or did ripgrep migrate to using io_uring already?
replies(1): >>44607385 #
burntsushi ◴[18 Jul 25 17:20 UTC] No.44607385{3}[source]▶
No, ripgrep doesn't use io_uring. Idk if it ever will.
replies(1): >>44608321 #
1. porridgeraisin ◴[18 Jul 25 18:41 UTC] No.44608321{4}[source]▶
Curious: Why? Is it not a good fit for what ripgrep does? Isn't the sort of "streaming" "line at a time" I/O that ripgrep does a good fit for async io?
replies(1): >>44608899 #
2. burntsushi ◴[18 Jul 25 19:35 UTC] No.44608899[source]▶
For many workloads, ripgrep spends the vast majority of its time searching through files.

But more practically, it would be a terror to implement. ripgrep is built on top of platform specific standard file system APIs. io_uring would mean a whole heap of code to work with a different syscall pattern in addition to the existing code pattern for non-Linux targets.

So to even figure out whether it would be worth doing that, you would need to do a whole bunch of work just to test it. And because of my first point above, there is a hard limit on how much of an impact it could even theoretically have.

Where I would expect this to help is to batch syscalls during directory tree traversal. But I have nonidea how much it would help, if at all.

replies(1): >>44610266 #
3. Sesse__ ◴[18 Jul 25 21:58 UTC] No.44610266[source]▶
I believe that io_uring does not support getdents (despite multiple patch series being proposed). So you'd get async stat(), if you need them, but nothing else.
replies(1): >>44610690 #
4. ◴[18 Jul 25 22:55 UTC] No.44610690{3}[source]▶