WebSockets cost us $1M on our AWS bill

>In a typical TCP/IP network connected via ethernet, the standard MTU (Maximum Transmission Unit) is 1500 bytes, resulting in a TCP MSS (Maximum Segment Size) of 1448 bytes. This is much smaller than our 3MB+ raw video frames.

> Even the theoretical maximum size of a TCP/IP packet, 64k, is much smaller than the data we need to send, so there's no way for us to use TCP/IP without suffering from fragmentation.

Just highlights that they do not have enough technical knowledge in house. Should spend the $1m/year saving on hiring some good devs.

I fail to see how TCP/IP fragmentation really affects this use case. I don't know why it's mentioned and given that there aren't multiple network devices with different MTUs it will cause issues. Am I right? Is that the lack of technical knowledge you're referring to or am I missing something?

Sounds weird that apparently they expected to send 3 MB in a single TCP packet

Please explain?

Why do you say that? Their solution of using shared memory (structured as a ring buffer) sounds perfect for their use case. Bonus points for using Rust to do it. How would you do it?

Edit: I guess perhaps you're saying that they don't know all the networking configuration knobs they could exercise, and that's probably true. However, they landed on a more optimal solution that avoided networking altogether, so they no longer had any need to research network configuration. I'd say they made the right choice.

Modern NICs will do that for you via a feature called TSO -- TCP Segmentation Offload.

More shocking to me is that anyone would attempt to run network throughput oriented software inside of Chromium. Look at what Cloudflare and Netflix do to get an idea what direction they should really be headed in.

Yes, maybe they're talking about this: https://en.wikipedia.org/wiki/TCP_window_scale_option

This reminds me of when I was first starting to learn “real game development” (not using someone else's engine)—I was using C#/MonoGame, and, while having no idea what I was doing, decided I wanted vector graphics. I came across libcairo, figured out how to use it, set it all up correctly and everything… and then found that, whoops, sending 1920x1080x4 bytes to your GPU to render, 60 times a second, doesn't exactly work—for reasons that were incredibly obvious, in retrospect! At least it didn't cost me a million bucks to learn from my mistake.

The article reads a like a personal "learn by doing" blog post.

They use Chromium (or any other browser) not out of choice but because they have to in order to participate in third party video conference sessions. Of course it’s best to reverse engineer the video conferencing clients and do HTTP requests directly without a headless browser, but I presume they’ve tried that and it’s very difficult, not to mention prone to breaking at any moment.

What’s surprising to me is they can’t access the compressed video on the wire and have to send decoded raw video. But presumably they’ve thought about that too.