It's time to replace TCP in the datacenter (2023)

(arxiv.org)

188 points ilove_banh_mi | 1 comments | 18 Nov 24 01:42 UTC | HN request time: 0.26s | source

Show context

UltraSane ◴[18 Nov 24 05:43 UTC] No.42170007[source]▶

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receivers to control how much data senders can send. I have worked with a lot of VMware clusters that use FC to connect servers to storage arrays and it has ALWAYS worked perfectly.

replies(9): >>42170384 #>>42170465 #>>42170698 #>>42171057 #>>42171576 #>>42171890 #>>42174071 #>>42174140 #>>42175585 #

wejick ◴[18 Nov 24 08:20 UTC] No.42170698[source]▶

>>42170007 #

I'm imagining having a shared memory mounted as block storages then do the RPC thru this block. Some synchronization and polling/notifications work will need to be done.

replies(2): >>42171599 #>>42171658 #

1. creshal ◴[18 Nov 24 11:47 UTC] No.42171599[source]▶

>>42170698 #

The literal version of this is used by sanlock et al. to implement cluster-wide locks. But the whole "pretending to be block devices" overhead makes it not exactly ideal for anything else.

Drop the "pretending to be block devices" part and you basically end up with InfiniBand. It works well, if you ignore the small problem of "you need to either reimplement all your software to work with RDMA based IPC, or reimplement Ethernet on top of InfiniBand to remain compatible and throw away most advantages of InfiniBand again".

↑