Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

Every time a big company screws up, there are two highly informed sets of people who are guaranteed to be lurking, but rarely post, in a thread like this:

1) those directly involved with the incident, or employees of the same company. They have too much to lose by circumventing the PR machine.

2) people at similar companies who operate similar systems with similar scale and risks. Those people know how hard this is and aren’t likely to publicly flog someone doing their same job based on uninformed speculation. They know their own systems are Byzantine and don’t look like what random onlookers think it would look like.

So that leaves the rest, who offer insights based on how stuff works at a small scale, or better yet, pronouncements rooted in “first principles.”

Right? A common complaint by outsiders is that Netflix uses microservices. I'd love to hear exactly how a monolith application is guaranteed to perform better, with details. What is the magic difference that would have ensured the live stream would have been successful?

I am one of the ones who complain about their microservices architecture quite a lot.

This comes from both first-hand experience of talking to several of their directors when consulted upon on how to make certain systems of theirs better.

It's not just a matter of guarantees, it's a matter of complexity.

Like right now Google search is dying and there's nothing that they can do to fix it because they have given up control.

The same thing happened with Netflix where they wanted to push too hard to be a tech company and have their tech blogs filled with interesting things.

On the back end they went too deep on the microservices complexity. And on the front end for a long time they suffered with their whole RxJS problem.

So it's not an objective matter of what's better. It's more cultural problem at Netflix. Plus the fact that they want to be associated with "Faang" and yet their product is not really technology based.

You can explain these problems with simple business metrics that technologists like to ignore. Right before the recent Twitter acquisition, the various bits of info that came to the limelight included the "minor detail" that they had more than doubled their headcount and associated expenses, but had not doubled either their revenue or profits. Technology complexity went up, the business went backwards. Thousands of programmers doesn't always translate to more value!

Netflix regularly puts out blog articles proudly proclaiming that they process exabytes of logs per microsecond or whatever it is that their microservices Rube Goldberg machine spits out these days, patting themselves on the back for a heroic job well done.

Meanwhile, I've been able to go on the same rant year after year that they're still unable to publish more than five subtitle languages per region. These are 40 KB files! They had an employee argue with me about this in another forum, saying that the distribution of these files is "harder than I thought".

It's not hard!

They're solving the wrong problems. The problems they're solving are fun for engineers, but pointless for the business or their customers.

From a customer perspective Netflix is either treading water or noticeably getting worse. Their catalog is smaller than it was. They've lost licensing deals for movies and series that I want to watch. The series they're producing themselves are not things I want to watch any more. They removed content ratings, so I can't even pick something that is good without using my phone to look up each title manually!

Microservices solve none of these issues (or make it worse), yet this is all we hear about when Netflix comes up in technology discussions. I've only ever read one article that is actually relevant to their core business of streaming video, which was a blog about using kTLS in BSD to stream directly from the SSD to the NIC and bypassing the CPU. Even that is questionable! They do this to enable HTTPS... which they don't need! They could have just used a cryptographic signature on their static content, which the clients can verify with the same level of assurance as HTTPS. Many other large content distribution networks do this.

It's 100% certain that someone could pretend to be Elon, fire 200-500 staff from the various Netflix microservices teams and then hire just one junior tech to figure out how to distribute subtitles... and that would materially improve customer retention while cutting costs with no downside whatsoever.

Yeah, try dealing with many frontends with mixed HTTP and HTTPS, it's a nightmare and won't always work. Additionally, you want security on content delivery for revenue protection reasons. The way you've massively over simplified the BSD work shows that you perhaps didn't understand what they did and why hardware offload is a good thing?

Subtitles are also complicated because you have to deal with different media player frameworks on the +40 different players you deal with. Getting those players, which you may not own, to recognise multiple sub tracks can be a PITA.

Things look simple to a junior developer, but those experience in building streaming platforms at scale know there are dragons when you get into the implementation. Sometimes developers and architects do over complicate things, but smart leaders avoid writing code, so its an assumption to say things are being made over complicated.