It's time to replace TCP in the datacenter (2023)

> I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter

But it is often used for block storage in datacenters. Using it for anything else is going to be hard, as it is incompatible with TCP.

The problem with not using TCP is the same thing HOMA will face - anything already speaks TCP, nearly all potential hires know TCP and most problems you have with TCP have been solved by smart engineers already. Hardware is also easily available. Once you drop all those advantages, either your scale or your gains need to be massive to make that investment worth it, which is why TCP replacements are so rare outside of FAANG.

I wonder if there are any work on making something similar ( conceptually ) to TCP, super / sub set of TCP while offering 50-80% benefits of HOMA.

I guess I am old. Everytime I see new tech that wants to be hyped, completely throw out everything that is widely supported and working for 80-90% of uses cases, not battle tested and may be conceptually complex I will simply pass.

If you have a sufficiently stable network and/or known failure cases, you can already tune TCP quite a bit with nodelay, large congestion windows etc.. There's also QUIC, which basically is a modern implementation of TCP on top of UDP (with some trade-offs chosen with HTTP in mind). Once you stray too far, you'll loose the ability to use off-the-shelve hardware, though, at which point you'll quickly hit the point of diminishing returns - especially when simply upgrading the speed of the network hardware is usually a cheap alternative.

But you might not need TCP. For example, using file-sockets between an app, db, and http server (rails+pgsql+nginx for example) has many benefits. The beauty of OSI layers.

That would work on a single host, but the context of the datacenter probably assumes multihost/manyhost workloads.

Unix sockets can use tcp, udp, or be a raw stream

https://en.wikipedia.org/wiki/Unix_domain_socket#:~:text=The....

Puma creates a `UnixServer` which is a ruby stdlib class, using the defaults, which is extending `UnixSocket` which is also using the defaults

https://github.com/puma/puma/blob/fba741b91780224a1db1c45664...

Those defaults are creating a socket of type `SOCK_STREAM`, which is a tcp socket

> SOCK_STREAM will create a stream socket. A stream socket provides a reliable, bidirectional, and connection-oriented communication channel between two processes. Data are carried using the Transmission Control Protocol (TCP).

https://github.com/ruby/ruby/blob/5124f9ac7513eb590c37717337...

You still have the tcp overhead when using a local unix socket with puma, but you do not have any network overhead.

QUIC feels very pragmatic in terms of being built on UDP. As a lay person I don’t have a sense what additional gains might be on the table if the UDP layer were also up for reconsideration.

UDP has very low cost, the header is pretty much source and dest ports. For this low, low price, you get compatibility with existing routing, firewalling, NAT, etc.

How do you think 'file-sockets' are implemented over a network?

Hey! I know it’s muddled but that’s not quite correct. SOCK_STREAM is more general than TCP; SOCK_STREAM just means the socket is a byte stream. You would need to add IPPROTO_TCP on top of that to pull in the TCP stack.

UDS using SOCK_STREAM does not do that; ie, it is not using IPPROTO_TCP.

Unix domain stream sockets do not use tcp. Nor do unix datagram sockets use udp. They're much simpler.

>The type parameter should be one of two common socket types: stream or datagram.[10] A third socket type is available for experimental design: raw.

> SOCK_STREAM will create a stream socket. A stream socket provides a reliable, bidirectional, and connection-oriented communication channel between two processes. Data are carried using the Transmission Control Protocol (TCP).

> SOCK_DGRAM will create a datagram socket.[b] A Datagram socket does not guarantee reliability and is connectionless. As a result, the transmission is faster. Data are carried using the User Datagram Protocol (UDP).

> SOCK_RAW will create an Internet Protocol (IP) datagram socket. A Raw socket skips the TCP/UDP transport layer and sends the packets directly to the network layer.

I don't claim to be an expert, I just have a certain confidence that I'm able to comprehend words I read. It seems you can have 3 types of sockets, raw, udp, or tcp.

https://en.wikipedia.org/wiki/Unix_domain_socket

One issue with QUIC in e.g. C is how heavyweight it feels to use compared to garden-variety BSD sockets (and that’s already not the most ergonomic of APIs). I haven’t encountered a QUIC library that didn’t feel like it would absolutely dominate a simple application in both code size and API-design pressure. Of course, for large applications that’s less relevant, but the result is that switching to QUIC gets perceived as a Big Deal, a step for when you’re doing Serious Stuff. That’s not ideal.

I’d love to just play with QUIC a bit because it’s pretty neat, but I always get distracted by this problem and end up reading the RFCs, which so far I haven’t had the patience to get through.

Interesting! The Wikipedia article is quite wrong here. SOCK_STREAM certainly doesn’t imply TCP in all cases. I see the source is the Linux Programming Interface book; quite likely someone’s interpretation of that chapter was just wrong when they wrote this article. It is a subtle topic.

Not the first time a Wikipedia article has been wrong. That one seems to be talking about IP sockets and local ones at the same time instead of focusing on local ones. Could definitely stand to be rewritten.

They don’t have to use TCP. The point was to use sockets as the abstraction layer and use another inter-connect instead of TCP/IP. That way you’ve easily replaced TCP in the datacenter without major changes to many applications

Oh...your argument is replacing TCP is the easy part. Gotcha. Sure.

Yeah, someone's gotten confused. SOCK_DGRAM and SOCK_STREAM imply TCP and UDP when using AF_INET sockets, but not when using AF_UNIX sockets, though unix domain sockets do often somewhat do an impression of a TCP socket (e.g. they will report a connection state in the same way as TCP). Reads and writes to unix domain sockets essentially amount to the kernel copying memory between different processes (interestingly, linux will also do the same for local TCP/UDP connections as well, as an optimisation. So the data never actually gets formatted into separate packets). This also accounts for some of the things you can do with unix sockets you can't do with a network protocol, like pass permissions and file descriptors across them.