Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

Every time a big company screws up, there are two highly informed sets of people who are guaranteed to be lurking, but rarely post, in a thread like this:

1) those directly involved with the incident, or employees of the same company. They have too much to lose by circumventing the PR machine.

2) people at similar companies who operate similar systems with similar scale and risks. Those people know how hard this is and aren’t likely to publicly flog someone doing their same job based on uninformed speculation. They know their own systems are Byzantine and don’t look like what random onlookers think it would look like.

So that leaves the rest, who offer insights based on how stuff works at a small scale, or better yet, pronouncements rooted in “first principles.”

Right? A common complaint by outsiders is that Netflix uses microservices. I'd love to hear exactly how a monolith application is guaranteed to perform better, with details. What is the magic difference that would have ensured the live stream would have been successful?

I am one of the ones who complain about their microservices architecture quite a lot.

This comes from both first-hand experience of talking to several of their directors when consulted upon on how to make certain systems of theirs better.

It's not just a matter of guarantees, it's a matter of complexity.

Like right now Google search is dying and there's nothing that they can do to fix it because they have given up control.

The same thing happened with Netflix where they wanted to push too hard to be a tech company and have their tech blogs filled with interesting things.

On the back end they went too deep on the microservices complexity. And on the front end for a long time they suffered with their whole RxJS problem.

So it's not an objective matter of what's better. It's more cultural problem at Netflix. Plus the fact that they want to be associated with "Faang" and yet their product is not really technology based.

You can explain these problems with simple business metrics that technologists like to ignore. Right before the recent Twitter acquisition, the various bits of info that came to the limelight included the "minor detail" that they had more than doubled their headcount and associated expenses, but had not doubled either their revenue or profits. Technology complexity went up, the business went backwards. Thousands of programmers doesn't always translate to more value!

Netflix regularly puts out blog articles proudly proclaiming that they process exabytes of logs per microsecond or whatever it is that their microservices Rube Goldberg machine spits out these days, patting themselves on the back for a heroic job well done.

Meanwhile, I've been able to go on the same rant year after year that they're still unable to publish more than five subtitle languages per region. These are 40 KB files! They had an employee argue with me about this in another forum, saying that the distribution of these files is "harder than I thought".

It's not hard!

They're solving the wrong problems. The problems they're solving are fun for engineers, but pointless for the business or their customers.

From a customer perspective Netflix is either treading water or noticeably getting worse. Their catalog is smaller than it was. They've lost licensing deals for movies and series that I want to watch. The series they're producing themselves are not things I want to watch any more. They removed content ratings, so I can't even pick something that is good without using my phone to look up each title manually!

Microservices solve none of these issues (or make it worse), yet this is all we hear about when Netflix comes up in technology discussions. I've only ever read one article that is actually relevant to their core business of streaming video, which was a blog about using kTLS in BSD to stream directly from the SSD to the NIC and bypassing the CPU. Even that is questionable! They do this to enable HTTPS... which they don't need! They could have just used a cryptographic signature on their static content, which the clients can verify with the same level of assurance as HTTPS. Many other large content distribution networks do this.

It's 100% certain that someone could pretend to be Elon, fire 200-500 staff from the various Netflix microservices teams and then hire just one junior tech to figure out how to distribute subtitles... and that would materially improve customer retention while cutting costs with no downside whatsoever.

> Right before the recent Twitter acquisition, the various bits of info that came to the limelight included the "minor detail" that they had more than doubled their headcount and associated expenses, but had not doubled either their revenue or profits.

Every tech company massively inflated their headcount during the leadup to the Twitter acquisition because money was free.

I interviewed at Meta in 2021 and asked an EM what he would do if given a magic wand to fix one problem at the company. His response: "I would instantly hire 10,000 more engineers."

Elon famously did the opposite and now revenue is down 80%.

Money was not "free".

Borrowing costs went to nearly zero. That's not the same thing. You have to repay the money, you just don't have to repay it with interest.

I would have assumed people generally know this, but everybody (and I do mean everybody) talks like they don't know this. I would like to assume that "money is free" is just a shorthand, buuuut... again... these arguments! People like that EM talk like it was literally free money raining from the sky that could be spent (gone!) without it ever having to be repaid.

If you watched any of the long-form interviews Musk gave immediately after the acquisition, he made the point that if he hadn't bailed out Twitter, it had maybe 3 months of runway left before imploding.

Doubling headcount without a clear vision of how that would double revenues is madness. It is doubly so in orgs like Twitter or Netflix where their IT was already over-complicated.

It's too difficult for me to clearly and succinctly explain all the myriad ways in which a sudden inrush of noobs -- outnumbering the old guard -- can royally screw up something that is already at the edge of human capability due to complexity. There is just no way in which it would help matters. I could list the fundamental problems with that notion for hours.

Companies weren’t issuing debt to pay for headcount. The reason market interest rates matter is that when interest rates are low, your company stock doesn’t have to have high returns to get investment. When these conditions exist, companies feel safer hiring people to invest in growth instead of saving to provide high shareholder returns.

I highly recommend everyone take a university-level financial instruments course. The math isn’t super hard, and it does a very good job of explaining how rational investors behave.