←back to thread

WebSockets cost us $1M on our AWS bill

(www.recall.ai)
257 points tosh | 4 comments | 06 Nov 24 18:50 UTC | HN request time: 0.001s | source
Show context
trollied ◴[06 Nov 24 21:16 UTC] No.42069524[source]▶
>In a typical TCP/IP network connected via ethernet, the standard MTU (Maximum Transmission Unit) is 1500 bytes, resulting in a TCP MSS (Maximum Segment Size) of 1448 bytes. This is much smaller than our 3MB+ raw video frames.

> Even the theoretical maximum size of a TCP/IP packet, 64k, is much smaller than the data we need to send, so there's no way for us to use TCP/IP without suffering from fragmentation.

Just highlights that they do not have enough technical knowledge in house. Should spend the $1m/year saving on hiring some good devs.

replies(5): >>42069956 #>>42070181 #>>42070248 #>>42070804 #>>42070811 #
1. karamanolev ◴[06 Nov 24 21:45 UTC] No.42069956[source]▶
I fail to see how TCP/IP fragmentation really affects this use case. I don't know why it's mentioned and given that there aren't multiple network devices with different MTUs it will cause issues. Am I right? Is that the lack of technical knowledge you're referring to or am I missing something?
replies(1): >>42069979 #
2. drowsspa ◴[06 Nov 24 21:47 UTC] No.42069979[source]▶
Sounds weird that apparently they expected to send 3 MB in a single TCP packet
replies(1): >>42070420 #
3. bcrl ◴[06 Nov 24 22:21 UTC] No.42070420[source]▶
Modern NICs will do that for you via a feature called TSO -- TCP Segmentation Offload.

More shocking to me is that anyone would attempt to run network throughput oriented software inside of Chromium. Look at what Cloudflare and Netflix do to get an idea what direction they should really be headed in.

replies(1): >>42071388 #
4. oefrha ◴[06 Nov 24 23:44 UTC] No.42071388{3}[source]▶
They use Chromium (or any other browser) not out of choice but because they have to in order to participate in third party video conference sessions. Of course it’s best to reverse engineer the video conferencing clients and do HTTP requests directly without a headless browser, but I presume they’ve tried that and it’s very difficult, not to mention prone to breaking at any moment.

What’s surprising to me is they can’t access the compressed video on the wire and have to send decoded raw video. But presumably they’ve thought about that too.