←back to thread

It's time to replace TCP in the datacenter (2023)

(arxiv.org)
188 points ilove_banh_mi | 3 comments | 18 Nov 24 01:42 UTC | HN request time: 0.001s | source
Show context
UltraSane ◴[18 Nov 24 05:43 UTC] No.42170007[source]▶
I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receivers to control how much data senders can send. I have worked with a lot of VMware clusters that use FC to connect servers to storage arrays and it has ALWAYS worked perfectly.
replies(9): >>42170384 #>>42170465 #>>42170698 #>>42171057 #>>42171576 #>>42171890 #>>42174071 #>>42174140 #>>42175585 #
holowoodman ◴[18 Nov 24 12:49 UTC] No.42171890[source]▶
Fibrechannel is far too expensive, you need expensive switches, cables/transceivers and cards in addition to the Ethernet you'll need anyways. And this Fibrechannel hardware is quite limited in what you can do with it, by far not as capable as the usual Ethernet/IP stuff with regards to routing, encryption, tunneling, filtering and what not.

Similar things are happening with stuff like Infiniband, it has become far too expensive and Ethernet/ROCE is making inroads in lower- to medium-end installations. Availability is also an issue, Nvidia is the only Infiniband vendor left.

replies(2): >>42171954 #>>42176658 #
1. UltraSane ◴[18 Nov 24 20:34 UTC] No.42176658[source]▶
Is there a fundamental reason why FC is more expensive than Ethernet?
replies(1): >>42177094 #
2. convolvatron ◴[18 Nov 24 21:12 UTC] No.42177094[source]▶
since its entire reason to exist was to effect an artificial market segmentation, I guess the answer is .. yes?
replies(1): >>42180658 #
3. UltraSane ◴[19 Nov 24 06:50 UTC] No.42180658[source]▶
It isn't an artificial market segmentation. Fibre Channel is a no compromise technology with a single purpose, to connect servers to remote storage with performance and reliability close to directly attached storage, and it does that really, REALLY well. It is by far the single most bullet proof technology I have ever used. In a parallel universe where FC won over ethernet and every Ethernet port in the world was an FC port I don't see why it would be any more expensive that ethernet.