←back to thread

io_uring is faster than mmap

(www.bitflux.ai)
283 points ghuntley | 5 comments | 04 Sep 25 22:01 UTC | HN request time: 0.001s | source
Show context
jared_hulbert ◴[04 Sep 25 23:18 UTC] No.45133330[source]▶
Cool. Original author here. AMA.
replies(5): >>45133433 #>>45133597 #>>45133666 #>>45133764 #>>45135337 #
1. john-h-k ◴[05 Sep 25 00:10 UTC] No.45133666[source]▶
You mention modern server CPUs have capability to “read direct to L3, skipping memory”. Can you elaborate on this?
replies(1): >>45133767 #
2. jared_hulbert ◴[05 Sep 25 00:27 UTC] No.45133767[source]▶
https://www.intel.com/content/www/us/en/io/data-direct-i-o-t...

AMD has something similar.

The PCIe bus and memory bus both originate from the processor or IO die of the "CPU" when you use an NVMe drive you are really just sending it a bunch of structured DMA requests. Normally you are telling the drive to DMA to an address that maps to the memory, so you can direct it cache and bypass sending it out on the DRAM bus.

In theory... the specifics of what is supported exactly? I can't vouch for that.

replies(1): >>45134306 #
3. josephg ◴[05 Sep 25 01:58 UTC] No.45134306[source]▶
I’d be fascinated to see a comparison with SPDK. That bypasses the kernel’s NVMe / SSD driver and controls the whole device from user space - which is supposed to avoid a lot of copies and overhead.

You might be able to set up SPDK to send data directly into the cpu cache? It’s one of those things I’ve wanted to play with for years but honestly I don’t know enough about it.

https://spdk.io/

replies(1): >>45134505 #
4. jared_hulbert ◴[05 Sep 25 02:39 UTC] No.45134505{3}[source]▶
spdk and I go way back. I'm confident it'd be about the same, possibly ~200-300MB/s more, I was pretty close to the rated throughput of the drives. Io_uring has really closed the gap that used to exist between the in kernel and userspace solutions.

With the Intel connection they might have explicit support for DDIO. Good idea.

replies(1): >>45139208 #
5. benlwalker ◴[05 Sep 25 14:45 UTC] No.45139208{4}[source]▶
SPDK will be able to fully saturate the PCIe bandwidth from a single CPU core here (no secret 6 threads inside the kernel). The drives are your bottleneck so it won't go faster, but it can use a lot less CPU.

But with SPDK you'll be talking to the disk, not to files. If you changed io_uring to read from the disk directly with O_DIRECT, you wouldn't have those extra 6 threads either. SPDK would still be considerably more CPU efficient but not 6x.

DDIO is a pure hardware feature. Software doesn't need to do anything to support it.

Source: SPDK co-creator