←back to thread

How to escape the Linux networking stack

(blog.cloudflare.com)
148 points meysamazad | 5 comments | 17 Nov 25 15:49 UTC | HN request time: 0s | source
Show context
notepad0x90 ◴[17 Nov 25 20:18 UTC] No.45957801[source]▶
I'm slightly surprised cloudflare isn't using a userspace tcp/ip stack already (faster - less context switches and copies). It's the type of company I'd expect to actually need one.
replies(2): >>45958128 #>>45959181 #
Droobfest ◴[17 Nov 25 20:51 UTC] No.45958128[source]▶
From 2016: https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp...
replies(1): >>45958213 #
notepad0x90 ◴[17 Nov 25 20:59 UTC] No.45958213[source]▶
Nice, they know better. But it also makes me wonder, because they're saying "but what if you need to run another app", I'd expect for things like loadbalancers for example, you'd only run one app per server on the data plane, the user space stack handles that, and the OS/services use a different control plane NIC with the kernel stack so that boxes are reachable even if there is link saturation, ddos,etc..

It also makes me wonder, why is tcp/ip special? The kernel should expose a raw network device. I get physical or layer 2 configuration happening in the kernel, but if it is supposed to do IP, then why stop there, why not TLS as well? Why run a complex network protocol stack in the kernel when you can just expose a configured layer 2 device to a user space process? It sounds like "that's just the way it's always been done" type of a scenario.

replies(3): >>45958565 #>>45959377 #>>45960224 #
hansvm ◴[18 Nov 25 00:57 UTC] No.45960224[source]▶
TCP/IP is, in theory (AFAIK all experiments related to this fizzled out a decade or two ago), a global resource when you start factoring in congestion control. TLS is less obviously something you would want kernel involvement from, give or take the idea of outsourcing crypto to the kernel or some small efficiency gains for some workloads by skipping userspace handoffs, with more gains possible with NIC support.
replies(2): >>45960346 #>>45961233 #
1. notepad0x90 ◴[18 Nov 25 04:00 UTC] No.45961233[source]▶
why can't it be global and user space? DNS resolution for example is done by user space, and it is global.
replies(1): >>45962879 #
2. 1718627440 ◴[18 Nov 25 08:53 UTC] No.45962879[source]▶
DNS isn't a shared resource, that needs to be managed and distributed fairly, among programs that don't trust and cooperate with each others.
replies(1): >>45969650 #
3. notepad0x90 ◴[18 Nov 25 17:52 UTC] No.45969650[source]▶
DNS resolution is a shared resource. The DNS client is typically a user-space OS service that resolves and caches DNS requests. What is resolved by one application is cached and reused by another. But at the app level, there are is no deconflicting happening like transport layer protocols. However, the same can be said about IP, IP addresses like name servers are configured system wide and shared by all apps.
replies(1): >>45969780 #
4. 1718627440 ◴[18 Nov 25 18:01 UTC] No.45969780{3}[source]▶
It can be shared access to a cache, but this is an implementation detail for performance reasons. There is no problem with having different processes resolve DNS with different code. There is a problem if two processes want to control the same IP address, or manage the same TCP port.
replies(1): >>45974584 #
5. notepad0x90 ◴[19 Nov 25 01:06 UTC] No.45974584{4}[source]▶
Yeah, but there is still no reason why an "ip_stack" process can't ensure a different IP isn't used and a "gnu_tcp" or whatever process can't ensure tcp ports are assigned to only one calling process. An exclusive lock on the raw layer 2 device is what you're looking for I think. I mean right now, applications can just open a raw socket and use a conflicting tcp port. I've done to kill TCP connections matching some criteria by sending the remote end an RST pretending to be the real process (legit use case). Which approach is more performant, secure, and resilient? that's the what i'm asking here.