←back to thread

It's time to replace TCP in the datacenter (2023)

(arxiv.org)
188 points ilove_banh_mi | 2 comments | 18 Nov 24 01:42 UTC | HN request time: 0.401s | source
Show context
UltraSane ◴[18 Nov 24 05:43 UTC] No.42170007[source]▶
I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receivers to control how much data senders can send. I have worked with a lot of VMware clusters that use FC to connect servers to storage arrays and it has ALWAYS worked perfectly.
replies(9): >>42170384 #>>42170465 #>>42170698 #>>42171057 #>>42171576 #>>42171890 #>>42174071 #>>42174140 #>>42175585 #
KaiserPro ◴[18 Nov 24 11:42 UTC] No.42171576[source]▶
FC was much more expensive than ethernet, so needed a reason to be used.

For block storage it is great, if slower than ethernet.

replies(1): >>42176553 #
1. UltraSane ◴[18 Nov 24 20:26 UTC] No.42176553[source]▶
Is their a fundamental reason for it being more expensive than Ethernet or is just greed?
replies(1): >>42183545 #
2. KaiserPro ◴[19 Nov 24 14:05 UTC] No.42183545[source]▶
At the time it was rarer and required more specialised hardware.

FC was one of the first non-mainframe specific storage area network fabrics. One of the key differences between FC and ethernet is the collision detection/avoidance and guaranteed throughput. All that extra coordination takes effort on big switches, so it cost more to develop/produce.

You could shovel data through it at "line speed" and know that it'll get there, or you'll get an error at the fabric level. Ethernet happily takes packets, and if it can't deliver, then it'll just drop them. (well, kinda, not all ethernet does that, because ethernet is the english language of layer 2 protocols)