←back to thread

How to escape the Linux networking stack

(blog.cloudflare.com)
148 points meysamazad | 2 comments | 17 Nov 25 15:49 UTC | HN request time: 0.001s | source
Show context
notepad0x90 ◴[17 Nov 25 20:18 UTC] No.45957801[source]▶
I'm slightly surprised cloudflare isn't using a userspace tcp/ip stack already (faster - less context switches and copies). It's the type of company I'd expect to actually need one.
replies(2): >>45958128 #>>45959181 #
Droobfest ◴[17 Nov 25 20:51 UTC] No.45958128[source]▶
From 2016: https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp...
replies(1): >>45958213 #
notepad0x90 ◴[17 Nov 25 20:59 UTC] No.45958213[source]▶
Nice, they know better. But it also makes me wonder, because they're saying "but what if you need to run another app", I'd expect for things like loadbalancers for example, you'd only run one app per server on the data plane, the user space stack handles that, and the OS/services use a different control plane NIC with the kernel stack so that boxes are reachable even if there is link saturation, ddos,etc..

It also makes me wonder, why is tcp/ip special? The kernel should expose a raw network device. I get physical or layer 2 configuration happening in the kernel, but if it is supposed to do IP, then why stop there, why not TLS as well? Why run a complex network protocol stack in the kernel when you can just expose a configured layer 2 device to a user space process? It sounds like "that's just the way it's always been done" type of a scenario.

replies(3): >>45958565 #>>45959377 #>>45960224 #
1. wmf ◴[17 Nov 25 21:31 UTC] No.45958565[source]▶
AFAIK Cloudflare runs their whole stack on every machine. I guess that gives them flexibility and maybe better load balancing. They also seem to use only one NIC.

why is tcp/ip special? The kernel should expose a raw network device. ... Why run a complex network protocol stack in the kernel when you can just expose a configured layer 2 device to a user space process?

Check out the MIT Exokernel project and Solarflare OpenOnload that used this approach. It never really caught on because the old school way is good enough for almost everyone.

why stop there, why not TLS as well?

kTLS is a thing now (mostly used by Netflix). Back in the day we also had kernel-mode Web servers to save every cycle.

replies(1): >>45960187 #
2. bbarnett ◴[18 Nov 25 00:51 UTC] No.45960187[source]▶
Was it Tux? I've only used it, a looong time ago, on load balancers.

https://en.wikipedia.org/wiki/TUX_web_server