Most active commenters
  • HumanOstrich(7)
  • esseph(4)

←back to thread

How to escape the Linux networking stack

(blog.cloudflare.com)
148 points meysamazad | 15 comments | 17 Nov 25 15:49 UTC | HN request time: 1.417s | source | bottom
Show context
marginalia_nu ◴[17 Nov 25 23:18 UTC] No.45959577[source]▶
This is extremely tangential, but I was working on setting up some manual network namespaces recently, basically manually reproducing what docker does to fix some of its faulty assumptions regarding containers having multiple IPs and a single name causing all sort of jank, and had to freshen up on a lot of Linux virtual networking concepts (namespaces, veths, bridge networks, macvlans and various other interfaces), made a ton of fairly informal notes to make myself sufficiently familiar with the thing to set it up.

Would anyone be interested if I polished it up and maybe added a refresher on the relevant layer 2 networking needed to reason about it? It's a fair bit of work and it's a niche topic, so I'm trying to poll a bit to see if the juice is worth the squeeze.

replies(11): >>45959749 #>>45959968 #>>45960118 #>>45960266 #>>45960554 #>>45960755 #>>45961911 #>>45961983 #>>45962002 #>>45962168 #>>45967111 #
1. HumanOstrich ◴[18 Nov 25 01:05 UTC] No.45960266[source]▶
I was actually going down rabbitholes today trying to figure out how to do a sane Docker setup where all the containers couldn't connect to each other. Your notes would be valuable at most any level of polish.
replies(2): >>45961588 #>>45966377 #
2. esseph ◴[18 Nov 25 05:06 UTC] No.45961588[source]▶
If you create each container in its own network namespace, they won't be able to.
replies(1): >>45961736 #
3. HumanOstrich ◴[18 Nov 25 05:35 UTC] No.45961736[source]▶
It's a little more complex than that for any non-trivial layout where some containers do need to talk to other containers, but most don't.
replies(2): >>45961964 #>>45968890 #
4. brirec ◴[18 Nov 25 06:22 UTC] No.45961964{3}[source]▶
You could also create a network for each pair of containers that need to communicate with one another.
replies(2): >>45962220 #>>45964993 #
5. HumanOstrich ◴[18 Nov 25 07:07 UTC] No.45962220{4}[source]▶
That would create an excessive amount of bridges in my case. Also this is another trivial suggestion that anyone can find with a quick search or asking an LLM. Not helpful.

I'm not sure why people are replying to my comment with solutioning and trivial suggestions. All I did was encourage the thread OP to publish their notes. FWIW I've already been through a lot of options for solving my issue, and I've settled on one for now.

replies(1): >>45962711 #
6. kortilla ◴[18 Nov 25 08:29 UTC] No.45962711{5}[source]▶
> I'm not sure why people are replying to my comment with solutioning and trivial suggestions

Because your comment didn’t say you solved it and you asked for notes without any polish as if that would help.

replies(1): >>45966206 #
7. marginalia_nu ◴[18 Nov 25 12:49 UTC] No.45964993{4}[source]▶
If you want point-to-point communication between two network namespaces, you should use veths[1]. I think virtual patch cables is a good mental model for veths.

If you want multiple participants, you use bridges, which are roughly analogous to switches.

[1] https://man7.org/linux/man-pages/man4/veth.4.html

8. HumanOstrich ◴[18 Nov 25 14:07 UTC] No.45966206{6}[source]▶
I didn't say I settled on a solution for all time. I said "for now". I'm still interested in alternatives.
9. aryonoco ◴[18 Nov 25 14:19 UTC] No.45966377[source]▶
I put each docker container in a LXC container which effectively uses namespaces, cgroups etc to isolate them.
10. esseph ◴[18 Nov 25 16:58 UTC] No.45968890{3}[source]▶
That's a change from what was asked which was isolation between each.

Yes, if they need to talk, share namespaces.

If you don't want a generic but true answer, don't ask a generic question and then be upset when the responses don't have enough detail about your specific situation that you hadn't described :-)

replies(1): >>45971922 #
11. HumanOstrich ◴[18 Nov 25 20:55 UTC] No.45971922{4}[source]▶
I didn't ask a question and I wasn't upset. :-)
replies(1): >>45979652 #
12. esseph ◴[19 Nov 25 14:06 UTC] No.45979652{5}[source]▶
If you need more / different isolation, you're going to need custom nftables/ebtables rules.

In another model you could drop each bridge onto a unique vlan, and firewall them.

There's tons of options out there.

Anyway, if you had more specifics to go off of, there's plenty of network engineers and kubernetes/docker admins floating around willing to help - maybe start a Ask HN post?

replies(1): >>45980376 #
13. HumanOstrich ◴[19 Nov 25 14:59 UTC] No.45980376{6}[source]▶
You're still offering suggestions I said I didn't ask for. I'm sure you're trying to help, but at this point you're coming across as passive-aggressive.
replies(1): >>45980470 #
14. esseph ◴[19 Nov 25 15:07 UTC] No.45980470{7}[source]▶
You asked for the notes of somebody that's done isolation in different ways in docker.

Your responses have confused me so much I showed them to my partner, who is also confused.

replies(1): >>45980610 #
15. HumanOstrich ◴[19 Nov 25 15:17 UTC] No.45980610{8}[source]▶
I asked the person I was replying to for their notes because they were asking if anyone was interested in them.