←back to thread

It's time to replace TCP in the datacenter (2023)

(arxiv.org)
188 points ilove_banh_mi | 2 comments | 18 Nov 24 01:42 UTC | HN request time: 0.459s | source
Show context
UltraSane ◴[18 Nov 24 05:43 UTC] No.42170007[source]▶
I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receivers to control how much data senders can send. I have worked with a lot of VMware clusters that use FC to connect servers to storage arrays and it has ALWAYS worked perfectly.
replies(9): >>42170384 #>>42170465 #>>42170698 #>>42171057 #>>42171576 #>>42171890 #>>42174071 #>>42174140 #>>42175585 #
1. markhahn ◴[18 Nov 24 16:38 UTC] No.42174071[source]▶
the question is really: does it have anything vendor-specific, interop-breakers?

FC seems to work nicely in a single-vendor stack, or at least among specific sets of big-name vendors. that's OK for the "enterprise" market, where prices are expected to be high, and where some integrator is getting a handsome profit for making sure the vendors match.

besides consumer, the original non-enterprise market was HPC, and we want no vendor lock-in. hyperscale is really just HPC-for-VM-hosting - more or less culturally compatible.

besides these vendor/price/interop reasons, FC has never done a good job of keeping up. 100/200/400/800 Gb is getting to be common, and is FC there?

resolving congestion is not unique to FC. even IB has more, probably better solutions, but these days, datacenter ethernet is pretty powerful.

replies(1): >>42176648 #
2. UltraSane ◴[18 Nov 24 20:34 UTC] No.42176648[source]▶
FC speeds have really lagged. 64Gbps is available but not widely used and 128Gbps was introduced in 2023. But since by definition 100% of FC bandwidth can only be used for storage it has been adequate.

https://fibrechannel.org/preview-the-new-fibre-channel-speed...