Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

Right? A common complaint by outsiders is that Netflix uses microservices. I'd love to hear exactly how a monolith application is guaranteed to perform better, with details. What is the magic difference that would have ensured the live stream would have been successful?

It's not guaranteed, but much fewer points of failure.

I doubt a "microservice" has anything to do with delivering the video frames. There are specific kinds of infrastructure tech that are specifically designed to serve live video to large amounts of clients. If they are in fact using a "microservice" to deliver video frames, then I'd ask them to have their heads examined. Microservices are typically used to do mundane short-lived tasks, not deliver video.

> It's not guaranteed, but much fewer points of failure.

Can you explain where this is relevant to buffering issues?

Also, you are very wrong regarding failure modes. The larger the service, the more failure modes it has. Moreover, in monoliths if a failure mode can take down/degrade the whole service, all other features are taken down/degraded. Is having a single failure mode that brings down the whole service what you call fewer points of failure?

There’s very likely a dedicated service for delivering frames.

That’s service would technically be a “microservice” even if it is a large service.

I can't, since I don't know Netflix's architecture - I was responding to "I'd love to hear exactly how a monolith application is guaranteed to perform better, with details."

I am one of the ones who complain about their microservices architecture quite a lot.

This comes from both first-hand experience of talking to several of their directors when consulted upon on how to make certain systems of theirs better.

It's not just a matter of guarantees, it's a matter of complexity.

Like right now Google search is dying and there's nothing that they can do to fix it because they have given up control.

The same thing happened with Netflix where they wanted to push too hard to be a tech company and have their tech blogs filled with interesting things.

On the back end they went too deep on the microservices complexity. And on the front end for a long time they suffered with their whole RxJS problem.

So it's not an objective matter of what's better. It's more cultural problem at Netflix. Plus the fact that they want to be associated with "Faang" and yet their product is not really technology based.

> they want to be associated with "Faang" and yet their product is not really technology based.

You lost me. Netflix built a massive CDN, a recommendation engine, did dynamic transcoding of video, and a bunch of other things, at scale, quite some years before everyone else. They may have enshittified in the last five years, but I dont see any reason why they dont have a genuinely legitimate claim to being a founder member of the FAANG club.

I have a much harder time believing that companies with AI in their name or domain are doing any kind of AI, by contrast.

You can explain these problems with simple business metrics that technologists like to ignore. Right before the recent Twitter acquisition, the various bits of info that came to the limelight included the "minor detail" that they had more than doubled their headcount and associated expenses, but had not doubled either their revenue or profits. Technology complexity went up, the business went backwards. Thousands of programmers doesn't always translate to more value!

Netflix regularly puts out blog articles proudly proclaiming that they process exabytes of logs per microsecond or whatever it is that their microservices Rube Goldberg machine spits out these days, patting themselves on the back for a heroic job well done.

Meanwhile, I've been able to go on the same rant year after year that they're still unable to publish more than five subtitle languages per region. These are 40 KB files! They had an employee argue with me about this in another forum, saying that the distribution of these files is "harder than I thought".

It's not hard!

They're solving the wrong problems. The problems they're solving are fun for engineers, but pointless for the business or their customers.

From a customer perspective Netflix is either treading water or noticeably getting worse. Their catalog is smaller than it was. They've lost licensing deals for movies and series that I want to watch. The series they're producing themselves are not things I want to watch any more. They removed content ratings, so I can't even pick something that is good without using my phone to look up each title manually!

Microservices solve none of these issues (or make it worse), yet this is all we hear about when Netflix comes up in technology discussions. I've only ever read one article that is actually relevant to their core business of streaming video, which was a blog about using kTLS in BSD to stream directly from the SSD to the NIC and bypassing the CPU. Even that is questionable! They do this to enable HTTPS... which they don't need! They could have just used a cryptographic signature on their static content, which the clients can verify with the same level of assurance as HTTPS. Many other large content distribution networks do this.

It's 100% certain that someone could pretend to be Elon, fire 200-500 staff from the various Netflix microservices teams and then hire just one junior tech to figure out how to distribute subtitles... and that would materially improve customer retention while cutting costs with no downside whatsoever.

Why is that “very likely”?

I’m genuinely curious about the reasoning behind that statement. It’s very possible that you are using a different set of assumptions or definitions than I am.

Google search is dying because of business reasons, not technical ones. The ads branch is actively cannibalizing search quality to make people perform more searches and view more ads.

"Microservices" have nothing to do with it.

> Right before the recent Twitter acquisition, the various bits of info that came to the limelight included the "minor detail" that they had more than doubled their headcount and associated expenses, but had not doubled either their revenue or profits.

Every tech company massively inflated their headcount during the leadup to the Twitter acquisition because money was free.

I interviewed at Meta in 2021 and asked an EM what he would do if given a magic wand to fix one problem at the company. His response: "I would instantly hire 10,000 more engineers."

Elon famously did the opposite and now revenue is down 80%.

Pornhub has better, more reliable technology than Netflix, yet you don't see their tech blog very often do you?

I say that because, for performance reasons, you’d never want to wait on potentially several hops to stream media and because the act of streaming could very well be a good enough domain boundary.

> I was responding to "I'd love to hear exactly how a monolith application is guaranteed to perform better, with details."

I asked nothing about Netflix. My question was directed at your remark regarding monoliths vs microservices.

Now, can you answer the question?

Money was not "free".

Borrowing costs went to nearly zero. That's not the same thing. You have to repay the money, you just don't have to repay it with interest.

I would have assumed people generally know this, but everybody (and I do mean everybody) talks like they don't know this. I would like to assume that "money is free" is just a shorthand, buuuut... again... these arguments! People like that EM talk like it was literally free money raining from the sky that could be spent (gone!) without it ever having to be repaid.

If you watched any of the long-form interviews Musk gave immediately after the acquisition, he made the point that if he hadn't bailed out Twitter, it had maybe 3 months of runway left before imploding.

Doubling headcount without a clear vision of how that would double revenues is madness. It is doubly so in orgs like Twitter or Netflix where their IT was already over-complicated.

It's too difficult for me to clearly and succinctly explain all the myriad ways in which a sudden inrush of noobs -- outnumbering the old guard -- can royally screw up something that is already at the edge of human capability due to complexity. There is just no way in which it would help matters. I could list the fundamental problems with that notion for hours.

fair.

I have always wondered how do they deliver their content and what goes on behind the scenes and nobody on tech twitter or even youtubers talk about pornhub's infra for some reason. A lot of the innovation in tech has roots in people wanting to see high quality tiddies on the internet.

Monoliths might be better in some cases, microservices in others. Performance can obviously differ based on implementation.

I'm fairly well versed in basic ip networks, but the above sounds like a wordsalad to me.

PH has to stream for 20 seconds on average while boxing match is a lot longer :-)

Twitter revive collapsed because of politics and the public removal of moderators, not be side of how many engineers were employed.

Same thing was done a little later by HBO, Disney and a plethora of others, which points to the task not really being uber-difficult.

Speaking as a consumer, Netflix’s solution is objectively better than it’s competitors. It handles network blips better, it’s more responsive to use, it has far fewer random bugs you need to work around.

You can argue whether or not that edge translates into more revenue, but the edge is objectively there.

Based on my experience as a client, they’re not as reliable, Disney in particular. I thought it was a solved problem but it’s not apparently.

I worked on PH infra for a few years -

* The frontend itself runs on bare metal

* A lot of the backend was built out as microservices, running on top of Mesos and then later K8s

* CDN was in-house. The long tail is surprisingly long, but the front page videos were cached globally

The unifying theme behind PH is that they host everything on their own equipment, no AWS/GCP involved.

That’s okay, you probably just haven’t worked with high performance services or micro services before.

Network requests (sometimes called hops) take a significant amount of time. You don’t want your streaming service to take a significant amounts of time.

In microservices land, you generally try making services based on some “domain” (metaphorical, not like a literal domain name) which defines the responsibility of any given service. Defining these domains is more art than science and depends on the business needs and each team.

Video streaming might be one of those domains for Netflix.

Companies weren’t issuing debt to pay for headcount. The reason market interest rates matter is that when interest rates are low, your company stock doesn’t have to have high returns to get investment. When these conditions exist, companies feel safer hiring people to invest in growth instead of saving to provide high shareholder returns.

I highly recommend everyone take a university-level financial instruments course. The math isn’t super hard, and it does a very good job of explaining how rational investors behave.

They want to be associated with FAANG? What does the N stand for?

So you’re saying the investors are happy to see their money set on fire?

Surely they expect at a minimum that their capital investment would make them dividends (increased revenue), and also that the money wasn’t simply set on fire with nothing to show for it and no way to repay it.

If I’m wrong then Twitter - and similar companies - are little better than Ponzi schemes, with investors relying on the money of the greater fool to recover their money.

Hard to speak "objectively" as a consumer who has their own regional biases and knows none of the sausage underneath.

Maybe you're in a rural area and Netflix scaled gracefully. Maybe you're deep in SF and Netflix simply outspent to give minimal disruption to a population hub. These could both be true but don't speak to what performs better overall.

Sounds like a consumer issue ;)

> So you’re saying the investors are happy to see their money set on fire?

Ah, HN, where you try to explain how things work, and you get ignorant sarcasm in return.

> Surely they expect at a minimum that their capital investment would make them dividends (increased revenue), and also that the money wasn’t simply set on fire with nothing to show for it and no way to repay it.

Yes, of course. But when safe investments (e.g., Treasuries) are paying out close to zero, investors are going to tolerate lower returns than they do when Treasuries are paying out 3% or more.

It's basic arithmetic: you take the guaranteed rate, add a risk premium, and that's what investors expect from riskier investments. This is well-covered in the class I recommended.

Also, not every investor thinks in terms of consistent return. A pensioner may have a need for a guaranteed 3% annual return to keep pace with inflation. A VC, on the other hand, is often content to have zero returns for years followed by a 100x payout through an IPO.

> A VC, on the other hand, is often content to have zero returns for years followed by a 100x payout through an IPO

I know how all this works, but 100x payout is for the small initial investments, not after 10 years of operating at multi-billion-dollar scales.

Small amounts of money are set on fire all of the time, chasing this kind of high-risk return.

Nonetheless, there's an expectation of a return, even if only in aggregate across many small startups.

What I was observing (from the outside, at a distance) was that Twitter was still being run by a startup despite being in an effectively monopoly position already and a "mature" company. Similarly, Amazon could set money on fire while they were the growing underdog. If they doubled their headcount today without doubling either revenue or profits, the idiots responsible for that would be summarily fired.

I get that Silicon Valley and their startup culture does a few things in an unusual way, but that doesn't make US dollars not be US dollars and magically turn into monopoly money that rains from the sky just because interest rates are low.

People here don't understand basic concepts like risk adjusted returns, flight to quality, Searching for yield etc...

Yeah, try dealing with many frontends with mixed HTTP and HTTPS, it's a nightmare and won't always work. Additionally, you want security on content delivery for revenue protection reasons. The way you've massively over simplified the BSD work shows that you perhaps didn't understand what they did and why hardware offload is a good thing?

Subtitles are also complicated because you have to deal with different media player frameworks on the +40 different players you deal with. Getting those players, which you may not own, to recognise multiple sub tracks can be a PITA.

Things look simple to a junior developer, but those experience in building streaming platforms at scale know there are dragons when you get into the implementation. Sometimes developers and architects do over complicate things, but smart leaders avoid writing code, so its an assumption to say things are being made over complicated.

Agree. Hulu, HBO/Max, and Disney Plus each do most of these:

- frequently decide that episodes I've watched are either completely unwatched (with random fully watched eps of the show mixed in).

- seemingly every time I leave at the start of the end-credits, I surely must have intended to come back and watch them.

- rebuild the entire interface (progressively, slowly) when I've left the tab unfocussed for too long. Instead of letting continue where I was, they show it for less than a second, the rebuild the world.

- keep resetting the closed-caption setting to "none", regardless of choosing "always" or "on instant replay"; worse, they sometimes still have the correct setting in the interface, but have disabled captions anyway.

Netflix has only once since they started streaming forgotten playback position or episode completion. They politely suggest when to reload the page (via a tiny footer banner), but even that might not appear for months. They usually know where end-credits really start, and count that as completion. They don't seem to mess with captions.

>"I would instantly hire 10,000 more engineers."

From my experience, this answer usually belies someone who doesn’t fully understand the system and problems of the business. The easy answer when overwhelmed is “we need more people.” To use a manufacturing analogy, you can cover up a lot of quality issues with increased throughout, but it makes for an awfully inefficient system.

Forced assoc.

Honestly, I've seen my fair share of 7 layer SOAP stacks. Not sure if any of your unsubstatiated handwaving is making any sense.

Tell me specifics.

> you perhaps didn't understand what they did

I read and understood their entire technical whitepaper. I get the what, I'm just saying that the why might not make as much sense as you might assume.

> +40 different players you deal with

They own the clients. They wrote the apps themselves. This is Netflix code reading data from Netflix servers. Even if there are third-party clients (wat!?), that doesn't explain why none of Netflix's home-grown clients support more than 5 subtitle languages.

> Getting those players, which you may not own, to recognise multiple sub tracks can be a PITA.

This is a core part of the service, which everyone else has figured out. Apple TV for example has dozens of subtitle languages.[1]

With all due respect: Read what you just wrote. You're saying that an organisation that has the engineering prowess to stream at 200 Gbps per edge box and also handles terabytes of diagnostic log ingestion per hour can't somehow engineer the distribution of 40 KB text files!?

I can't even begin to outline the myriad ways in which these excuses are patent nonsense.

These are children playing with the fun toys, totally ignoring like... 1/3rd of the viewing experience. As far as the users are concerned, there's nothing else of consequence other than the video, audio, and text that they see on the screen.

"Nah, don't worry about the last one, that only affects non-English speakers or the deaf, we only care about DEI for internal hires, not customers."

[1] Just to clarify: I'm asking for there to be an option to select one language at a time from all available languages, not showing multiple languages at once, which is a tiny bit harder. But... apparently not that hard, because I have two different free, open-source video players on my PC that can do this so I can have my spouse get "full" subtitles in a foreign language while I see the "auto" English subtitles pop up in a different colour when appropriate. With Netflix I have to keep toggling between her language and my language every time some foreign non-English thing is said. Netflix is worth $362B, apparently, but hasn't figured out something put together by poor Eastern European hobbyists in their spare time.

See, you're confused because you think that the media player is owned by Netflix.

The browser gives you a certain level of control on computers, although you have to deal with the oddities of Safari, but when you go to smart TVs it's the wild west. Netflix does provide their tested framework to TV vendors but it's still not easy, because media playback often requires hardware acceleration, but the rendering framework isn't standard.

Developing for set-top boxes, multiple generations of phones, and smart TVs comes with all sorts of oddities. You think it's easy because you haven't done it.