Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

When you step back and look at the situation, it's not hard to see why Netflix dropped the ball here. Here's now I see it (not affiliated with Netflix, pure speculation):

- Months ago, the "higher ups" at Netflix struck a deal to stream the fight on Netflix. The exec that signed the deal was probably over the moon because it would get Netflix into a brand new space and bring in large audience numbers. Along the way the individuals were probably told that Netflix doesn't do livestreaming but they ignored it and assumed their talented Engineers could pull it off.

- Once the deal was signed then it became the Engineer's problem. They now had to figure out how to shift their infrastructure to a whole new set of assumptions around live events that you don't really have to think about when streaming static content.

- Engineering probably did their absolute best to pull this off but they had two main disadvantages, first off they don't have any of the institutional knowledge about live streaming and they don't really know how to predict demand for something like this. In the end they probably beefed up livestreaming as much as they could but still didn't go far enough because again, no one there really knows how something like this will pan out.

- Evening started off fine but crap hit the fan later in the show as more people tuned in for the main card. Engineering probably did their best to mitigate this but again, since they don't have the institutional knowledge of live events, they were shooting in the dark hoping their fixes would stick.

Yes Netflix as a whole screwed this one up but I'm tempted to give them more grace than usual here. First off the deal that they struck was probably one they couldn't ignore and as for Engineering, I think those guys did the freaking best they could given their situation and lack of institutional knowledge. This is just a classic case of biting off more than one can chew, even if you're an SV heavyweight.

>First off the deal that they struck was probably one they couldn't ignore

If you can't provide the service you shouldn't sell it?

> They now had to figure out how to shift their infrastructure to a whole new set of assumptions around live events

It wasn't their first live event. A previous live event had similar issues.

You've never worked in a startup have you? Or any business for that matter. You have to promise something first, then build it.

My speculation here is this was just classic SV cockiness. The team that closed this deal probably knew that they didn't have the capability but I'm sure the arguments for doing it anyways was something along the lines of: "we have the best engineers in the bay area, we can probably figure this out"

There are endless amounts of stories and situations in which selling something before it really exists has helped businesses. It's totally plausible that a team working on video streaming at the scale of Netflix could figure out live streaming.

Pre-optimization is definitely a thing and it can massively hurt (i.e. startups go under) businesses. Let's stop pretending any businesses would say 'no' to extra revenue even before the engineering team had full assurance there was no latency drop.

This isn't Netflix's first foray into livestreaming. They tried a livestream last year for a reunion episode of one of their reality TV shows which encountered similar issues [0]. Netflix already has a contract to livestream a football event on Christmas, so it'll be interesting to see if their engineers are able to get anything done in a little over a month.

These failures reflect very poorly on Netflix leadership. But we all know that leadership is never held accountable for their failures. Whoever is responsible for this should at least come forward and put out an apology while owning up to their mistakes.

[0] https://time.com/6272470/love-is-blind-live-reunion-netflix/

Execs never listen or even ask engineers about feasibility of projects they sign up to. Hope the exec in question will be let go.

No joke, is this actually true?

Do startups really do this? I thought the capability is built or nearly built or at least in testing already with reasonable or amazing results, THEN they go to market?

Do startups go to other startups, fortune 500 companies and public companies to make false promises with or without due diligence and sign deals with the knowledge that the team and engineers know the product doesn't have the feature in place at all?

In other words:

Company A: "We can provide web scale live streaming service around the world to 10 billion humans across the planet, even the bots will be watching."

Company B: "OK, sounds good, Yes, here is a $2B contract."

Company A: "Now team I know we don't have the capability, but how do we build, test and ship this in under 6 months???"

I mean, the ones that do ask don’t proceed to signing up. I think we are seeing a form of survival bias.

Many do, as far as initial investment goes. It makes sense when you think about the capital intensive nature of most startups (including more than web startups here, e.g. lab tech commercialization). It also accurately describes a research grant.

That's for startups that can't bootstrap (most of them). For ones which can, they may still choose to do this with customers, as you describe, because it means letting their work follow the money.

“Aspirational sugar” is as common in startup culture as in Fortune 500 sales contracts, they’re just messaged and “de-risked” differently.

>> First off the deal that they struck was probably one they couldn't ignore

> If you can't provide the service you shouldn't sell it?

Then how will the folks in Sales get their commission?

Besides, not providing the service hasn't stopped Tesla from selling FSD, and their stock has been going gangbusters.

/s

Startups absolutely sell things they haven't made yet and might not even be capable of doing.

Next thing you know it's 9pm on a Sunday night and your desperately trying to ship a build for a client.

Netflix isn't some scrappy company though. If I had to guess they threw money at the problem.

A much better approach would of been to slowly scale over the course of a year. Maybe stream some college basketball games first, slowly picking more popular events to get some real prod experience.

Instead this is like their 3rd or 4th live stream ever. Even a pre show a week before would of allowed for greater testing.

I'm not a CTO of a billion dollar company though. I'm just an IC who's seen a few sites go down underload.

To be fair no one knows how it's going to go before it happens. It would of been more surprising for them to pull this off without issues... It's a matter of managing those issues. I know if I had paid 30$ for a Netflix subscription to watch this specific event I'd assume I got ripped off.

Livestreaming is a solved problem. This sounds like NIH [1]. (At the very least, hire them as a back-up.)

[1] https://en.wikipedia.org/wiki/Not_invented_here

If anything, startups are more transparent about it.

In the enterprise sector this is rampant. Companies sell "platforms" and those missing features are supposed to be implemented by consultants after the sale. This means the buyer is the one footing the bill for the time spent, and suffering with the delays.

Saying live-streaming is a solved problem is like saying search is a solved problem.

Look. I'm a small startup employee. I have a teeny tiny perspective here. But frankly speaking the idea that Netflix could just take some off the shelf widget and stuff it in their network to solve a problem... It's an absurd statement for even me. And if there's anyone it should apply to it would be a little startup company that needs to focus on their core area.

Every off the shelf component on the market needs institutional knowledge to implement, operate, and maintain it. Even Apple's "it just works" mantra is pretty laughable in the cold light of day. Very rarely in my experience do you ever get to just benefit from someone else's hard work in production without having an idea how properly implement, operate, and maintain it.

And that's at my little tiny ant scale. To call the problem of streaming "solved" for Netflix... Given the guess of the context from the GP post?

I just don't think this perspective is realistic at all.

You don't necessarily have to make false promises.

You can be totally honest and upfront that the functionality doesn't exist yet and needs to be built first, but that you think you understand the problem space and can handle the engineering, provided you can secure the necessary funding, where, by the way, getting a contract and some nominal revenue now could greatly help make this a reality...

And if the upside sounds convincing enough, a potential customer might happily sign up to cover part of your costs so they can be beta testers and observe and influence ongoing development.

Of course it happens all the time that the problem space turns out to be more difficult than expected, in which case they might terminate the partnership early and then the whole thing collapses from lack of funding.

i imagine this is why a lot of products, and startups, fail.

> Saying live-streaming is a solved problem is like saying search is a solved problem

It is. You can hire the people who have solved it to do it for you.

> the idea that Netflix could just take some off the shelf widget and stuff it in their network to solve a problem

Right. They have to hire one of the companies that does this. Each of YouTube, Twitch (Amazon), Facebook and TikTok have, I believe, handled 10+ million streams. The last two don't compete with Netflix.

Not sure why Netflix is held in high regard - this proves they're just as much clowns as the other 'big players' in the circus.

They arent clowns at all. Ita a totally different engineering problem and you cant just spin up live streaming capacity on demand. The entire system end to end isnt optimized for live streams yet.

That’s uncharitable. Proposing reasons for institutional failure and discussing those can be ways for humans to improve communication and said challenges.

> It is. You can hire the people who have solved it to do it for you.

"GPGPU compute is a solved problem if you buy Nvidia hardware" type comment

I mean, maybe? You just made all this up.

> "GPGPU compute is a solved problem if you buy Nvidia hardware" type comment

You're replacing the word hire with buy. That misconstrues the comment. If you need to do GPGPU compute and have never done it, you work with a team that has. (And if you want to build it in house, you scale to it.)

I believe this is the spirit of the "solved problem" comment: not that the solution is an off-the-shelf widget, but that if it has ever been solved, then that solution could technically be used again, even if organizing the right people is exorbitantly expensive.

Offering it for sale != having solved it.

There are multiple companies that offer this capability today that would take a few weeks to hide behind company branding. This was a problem of netflix just not being set up for live stream but thinking they could handle it.

>"GPGPU compute is a solved problem if you buy Nvidia hardware"

Which is valid? If your problem can be solved by writing a check, then it's the easiest problem to have on the planet.

Netflix didn't have to put out 3 PhD dissertations on how to improve the SOTA of live streaming, they only needed to reliably broadcast a fight for a couple hours.

That is a solved problem.

Amazon and Cloudflare do that for you as a service(!). Twitch and YouTube do it literally every day. Even X started doing it recently so.

No excuses for Netflix, tbh.

It's a way to mislead people into misunderstanding reality and therefore solving the wrong problems, often causing harm, now and in the future.

That's why serious analysis requires a factual basis, such as science, law, and good engineering and management. You need analytics data to figure out where the performance and organizational bottlenecks are.

Before people tried to understand illness with a factual basis, they wrote speculative essays on leeching and finding 'better' ways to do it.

> But we all know that leadership is never held accountable for their failures.

You've never heard of a CEO or other C-suite or VP getting fired?

It most definitely happens. On the other hand, people at every level make mistakes, and it's preferable that they learn from them rather than be fired, if at all possible.

Accountability can take many forms. I don't think they should be fired for making a mistake, I think they should release a statement recognizing their failure along with a post-mortem. Not a particularly high bar, but most leadership failures are often swept under the rug without any public accountability or evidence that they've learned anything.

We have evidence of prior failures with livestreaming from Netflix. Were the same people responsible for that failure or do we have evidence of them having learned anything between events? If anything, I'd expect the best leaders would have a track record that includes failures while showcasing their ability to overcome and learn from those mistakes. But based on what information is publicly available, this doesn't seem to be the case in this situation.

"Solved" merely means you don't need to invent something new to solve it. It doesn't mean trivial nor easy. And it definitely doesn't mean the problem is above trade-offs.

Landing on Mars is a solved problem. Nuclear bombs are a solved problem. Doesn't mean anyone can just write a check and get it done and definitely doesn't mean any business model can bear that cost.

Of course it means that!

You only need a big enough check.

No it doesn't.

India has landed on Mars for a fraction of the cost it took other nations, and the ESA has never been able to pull it off.

Not every cost is fungible and money isn't always the limiting factor.

It should be obvious that not all risks can be converted into a capital problem.

People say this, but then fall in love, get divorced, get depressed, or their company might lose its mojo, get sued, or lose an unreplaceable employee. But they will still say “all risk can be costed.”

At 120m concurrents? I’d be interested who can whitelabel that.

> Not every cost is fungible and money isn't always the limiting factor

Sure. This isn’t relevant to Netflix.

>India has landed on Mars for a fraction of the cost it took other nations, and the ESA has never been able to pull it off.

This further confirms my assertion, btw.

If it is just a matter of paying up, why hasn't ESA pulled it off? I'm pointing out and offering examples that "solved problem" has no bearing on the ease or organization capacity of any one group to do it. It is merely a statement that no unknown, new solution needs be invented.

If I have to spell it out you're clearly debating in bad faith and we're done here.

It is. The fact that 'streaming is a solved problem' has no bearing on any one company's ability to do it at scale. Solved problem means merely you don't have to invent something new, not that it is easy or within reach of everyone.

We are arguing if it's possible or not.

Who cares if a thousand guys are incapable? (like Netflix, lmao)

What matters are the ones that can do it, and you even said they've done it at "a fraction of the cost".

Paraphrasing, your argument says more about the incompetence of the ESA than the impossibility of doing such thing.

We now know it was more than 60m streams. I think it’s either a record or approaching one for a live stream.

In my language we have a saying that roughly translates to "Don't sell the hide until you've shot the bear".

And sure, there have probably been lots of examples where a business made promises they weren't confident about and succeeded. But there are surely also lots of examples where they didn't succeed.

So what's the moral of the story? I don't know, maybe if you take a gamble you should be prepared to lose that gamble. Sounds to me like Netflix fucked up. They sold something they couldn't provide. What are the consequences of that? I don't know, not do I particularly care. Not my problem.

Wait until you figure out how fortune 500 corporate vendors usually engage in the game of RFP box checking.