Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

People just do not appreciate how many gotchas can pop up doing anything live. Sure, Netflix might have a great CDN that works great for their canned content and I could see how they might have assumed that's the hardest part.

Live has changed over the years from large satellite dishes beaming to a geosat and back down to the broadcast center($$$$$), to microwave to a more local broadcast center($$$$), to running dedicated fiber long haul back to a broadcast center($$$), to having a kit with multiple cell providers pushing a signal back to a broadcast center($$), to having a direct internet connection to a server accepting a live http stream($).

I'd be curious to know what their live plan was and what their redundant plan was.

> People just do not appreciate how many gotchas can pop up doing anything live.

Sure thing, but also, how much resources do you think Netflix threw on this event? If organizations like FOSSDEM and CCC can do live events (although with way smaller viewership) across the globe without major hiccups on (relatively) tiny budgets and smaller infrastructure overall, how could Netflix not?

It is weird because this was a solved problem.

Every major network can broadcast the Super Bowl without issue.

And while Netflix claims it streamed to 280 million, that’s if every single subscriber viewed it.

Actual numbers put it in the 120 million range. Which is in line with the Super Bowl.

Maybe Netflix needs to ask CBS or ABC how to broadcast

Scale changes everything, I don't think it's fair to shrug this off

Do you live stream the superbowl? Me and everyone I know watch it over antenna broadcast tv. I think it is easier to have millions of tvs catch airwaves vs millions of point to point https video streams.

This is the whole point of chaos engineering that was invented at Netflix, which tests the resiliency of these systems.

I guess we now know the limits of what "at scale" is for Netflix's live-streaming solution. They shouldn't be failing at scale on a huge stage like this.

I look forward to reading the post mortem about this.

This is true, but scale comes after production. Once you have the video encoded on a server with a stable connection the hard part is over. What netflix failed to do is spread the files to enough servers around the globe to handle the load. I'm surprised they were unable(?) to use their network of edge servers to handle the live stream. Just run the stream with a 10 second delay and in that time push the stream segments to the edge server

Yeah, I agree with this, especially the everything part. Netflix isn't exactly a scrappy FOSS/hackers organization or similar.

This right here is where I'd expect the failure to occur. This isn't Joey Beercan running OBS using their home internet connectivity.

This is a major broadcast. I'd expect a full on broadcast truck/trailer. If they were attempting to broadcast this with the ($) option directly to a server from onsite, then I would demand my money back. Broadcasting a live IP signal just falls on its face so many times it's only the cheap bastard option. Get the video signal as a video signal away from the live location to a facility with stable redundant networking.

This is the kind of thinking someone only familiar with computers/software/networking would think of rather than someone in broadcasting. It's nice to think about disrupting, but this is the kind of failure that disruptors never think about. Broadcasters have been there done that with ensuring live broadcasts don't go down because an internet connection wasn't able to keep up.

If you watch it over cable, you're live streaming it. Let's face it, that's where the vast majority of viewers see it. Few people view OTA even if the quality is better.

Live sports do not broadcast the event directly to a streamer. They push it to their broadcast centers. It then gets distributed from there to whatever avenues it needs to go. Trying to push a live IP stream directly from the remote live venue rarely works as expected. That's precisely why the broadcasters/networks do not do it that way

Everyone keeps mentioning at scale. I seriously doubt this was an "at scale" problem. I have strong suspicion this was a failure at the origination point being able to push a stable signal. That is not an "at scale" issue, but a hubris of we can do better/cheaper than broadcasting standard practices

> how much resources do you think Netflix threw on this event?

Based on the results, I hope it was a small team working 20% time on the idea. If you tell me they threw everything they had at it to this result, then that's even more embarrassing for them.

> If organizations like FOSSDEM and CCC can do live events (although with way smaller viewership) […]

Or, for that matter, Youtube (Live) and Twitch.

> If you watch it over cable, you're live streaming it.

Which is probably done over the cableco's private network (not the public Internet) with a special VLAN used for television (as opposed to general web access). They're probably using multicast.

If it was a problem at origin, why did it get better/worse as viewership fell/rose?

As counterpoint, I observed 2-3 drops in bitrate, but an otherwise fine experience. So the problem seems to have been in dissemination, not at the origin.

> Once you have the video encoded on a server with a stable connection the hard part is over.

The hard part is over, and people new to the problem think they are almost done, but then the next part turns out to be 100x harder.

Lots of people can encode a video.

Last I checked, p2p solves a lot of the scaling issues.

Haven't Asian live sports been using p2p already two decades ago ?

(What is the biggest Peertube livestream so far ?)

When Netflix started it was the first in the space and breaking ground which is how they became a "tech" company that happens to stream media however it has been 15 years and since than the cloud providers have basically build "netflix as a service". I suspect most of the big streamers are using that instead of building their own in house thing and going through all the growing pains netflix is.

Lumen has their Vyvvyx product/service which uses fiber for broadcast television.

The CCC video crew has its fair share of geeks from broadcasting corporations and studio houses. Their combined institutional knowledge about live events and streaming distribution is probably in the same ballpark as that of giant global TV networks.

They also have the benefit of having practiced their craft at the CCC events for more than a decade. Twice a year. (Their summer event is smaller but still fairly well known. Links to talks show up on HN every now and then.)

Funky anecdote: the video crew at Assembly have more broadcasting and live AV gear for their annual event than most medium-sized studios.

You know they were commoditized when "Build Netflix" became a system-design interview question.

Solves differently though, right? Cable broadcasts are not the same as a streaming video over the internet, right?

Is cable video over IP now? Last time I looked (which was forever ago), even switched video was atsc with a bit of messaging for the cable box to ask what channel to tune to, and to keep the stream alive. TV over teleco systems seems to be highly multicast, so kind of similar, headend only has to send the content once, in a single bitrate.

Not really the same as an IP service live stream where the distribution point is sending out one copy per viewer and participating in bitrate adaptation.

AFAIK, Netflix hasn't publicly described how they do live events, but I think it's safe to assume they have some amount of onsite production that outputs the master feed for archiving and live transcoding for the different bitrate targets (that part may be onsite, or at a broadcast center or something cloudy), and then goes to a distribution network. I'd imagine their broadcast center/or onsite processing feeds to a limited number of highly connected nodes that feed to most of their CDN nodes; maybe more layers. And then clients stream from the CDN nodes. Nobody would stream an event like this direct from the event; you've got to have something to increase capacity.

You are making excuses for a multibillion dollar company that has been in this game for many years. Maybe the first to market in streaming.

This isn’t NFLX’s first rodeo in live streaming. Have seen a handful of events pop up in their apps.

There is no excuse. All of the resources and talent at their disposal, and they looked absolutely amateurish. Poor optics.

I would be amazed if they are able to secure another exclusive contract like this in the future.

A company that readily admits it burns out SWRs and SREs in exchange for the big bucks.

Just what the fuck are these people doing?

If I were a major investor in them I'd be pissed.

You’re talking about the contribution from the venue to the boardcast centre, increasingly not a full program but being mixed remotely.

That’s a very different area to transmission of live to end users.

> Is cable video over IP now?

Over the US and Canada it mostly is, though how advanced the transition is is very regional.

The plan is to drop both analog signal and digital (QAM) to reclaim the frequencies and use them for DOCSIS internet.

Newer set top boxes from Comcast (xfinity) runs over the internet connection (in a tagged VLAN on a private network, and they communicate over a hidden wifi).

I highly doubt this. Netflix has a system of OCAs that are loaded with hard disks, are installed in ISP’s networks, and serve the majority of those ISP’s customers.

Given than many people had no problems with the stream, it is unlikely to have been an origin problem but more likely the mechanism to fanout quickly to OCAs. Normally latency to an OCA doesn’t matter when you’re replicating new catalogs in advance, but live streaming makes a bunch of code that previously “didn’t need to be fast” get promoted to the hot path.

Sorry for the off topic but what’s this thing that I only come across in Hacker News about referring to a company by their stock exchange name (APPL, MSFT, etc) outside of a stock context? It seems really weird to me.

merriam-webster.com/dictionary/brevity

I've tried to watch an old Seinfeld episode during this event. It was freezing every few minutes even at downgraded bitrate. A video that should be on my local CDN node.

Writing APPL instead of Apple doesn’t get you any fewer keystrokes.

I am not sure that it is an issue with the origination point. In fact I just thought it was my ISP because my daughter's boyfriend was watching and doing facetime with her and my video was dropping but his was not. I have 2gb fiber and we regularly stream five TVs without any issue, so it should not have been a bandwidth issue.

also, the symbol for apple is AAPL.

I’ve been using vyvx since it was called global crossing/genesis, it was fairly unique when it started, but point to point ip distributon of programs has been the norm for at least 15 years. Still have backup paths on major events on a different technology, you’d be surprised how common a dual failure on two paths can be. For example output from the euro football this summer my mai paths were on a couple of leased lines with -7, but still had a backup on some local internet into a different city just incase there was a meltdown of the main providers network (it’s happened before with ipath, automation is great until it isn’t)

I just assume it's people who spend too much time thinking about the stock context.

Ugh. Similar (huge) pet peeve about people who say "n.B." instead of "note".

I have always assumed that a focus on stock tickers is the natural result when your primary user base is a group of people hyper focused on “total compensation” and stock grants. The name hackernews is merely a playful reference to the history of the site. Like the name “Patriot Act.”

In-group signaling for people who like playing or thinking about the stock market. Similar to how people who make travel a big part of their identity refer to cities by their airport code.

Is multicast a thing on the commercial internet? Seems like that could help.

> If you watch it over cable, you're live streaming it.

Those are multicast feeds.

> Trying to push a live IP stream directly from the remote live venue rarely works as expected.

In my experience it almost always works as expected. We have highly specialized codecs and equipment for this. The stream is actively managed with feedback from the receiver so parameters can be adjusted for best performance on the fly. Redundant connections and multiple backhauls are all handled automatically.

> That's precisely why the broadcasters/networks do not do it that way

We use fixed point links and satellite where possible because we own the whole pipe. It's less coordination and effort to setup and you can hit venues and remotes where fixed infrastructure is difficult or impossible to install.

> Their combined institutional knowledge about live events and streaming distribution

Now if they could just get audio levels and compression figured out.

Technically it’s one fewer keystroke (and it’s AAPL).

It’s a lot fewer keystrokes for MS (Morgan Stanley), GS (Goldman Sachs) and MSFT (Microsoft) than it is for AAPL, but it’s a force of habit for some. Once you’re used to referring to firms by their ticker symbols, you do it all the time.

E.g. an ex trader friend still says “spot” instead of “point” when referring to decimal points, even if talking in other contexts like software versions.

As a counterpoint to some other replies, I do this sometimes, not thinking at all about stocks but instead as a standardized abbreviation of sorts. Ms for example can mean tons of things from a title to multiple sclerosis to milliseconds. MSFT is clear and half the length.

If commercial = public, then no - you can not use multicast for this. It is heavily used within some enterprise networks though like if you go to a gym with lots of TVs they are all likely on multicast

Perhaps it was, or perhaps it was not.

I was watching a pirated, live retransmission of the event on Twitch (in Portuguese), and there was zero buffering on my end.

Tangent: Not fewer like F (Ford) or H (Hyatt Hotels). Unfortunately we don't have a full alphabet, missing {I, N, P, Q, Y}.

I guess if we allowed the first two letter ticker symbol for the missing singles, we could send messages by mentioning a bunch of company names.

Eg "Buy: Dominion Energy, Agilent. Hold: Nano Labs. Sell: Genpact." would refer to our esteemed moderator, and "Hyatt Hotels is pleased to announce special corporate rates for Nano Labs bookings" to this site itself.

[maybe it would be better to use the companies where the corporate name and ticker letter don't match? Like US Steel for X and AT&T for T?]

`Ms` would be megaseconds ;)

Which is why we started calling it M$ in the 2000s, emphasizing its overriding goal of making ca$h off its users

Um, aksually…

I was pointing out how dumb a multibillion dollar company is for getting this so wrong. Broadcasting live events is something that is underestimated by everyone that has never it, yet hubris of a major tech company thinking it knows better is biting them in the ass.

As many other people have commented, so many other very large dwarfing this event have been pulled off with no hiccups visible to the viewers. I have amazing stories of major hiccups during MLB World Series that viewers had no idea about happening, but “inside baseball” people knew. To the point that the head of the network caught something during the broadcast calling the director in the truck saying someone is either going to be fired or get a raise yet the audience would never have noticed if the person ended up getting fired. They didn’t, btw.

Or maybe they are pilots?

What are you talking about? The signal coming from a live event is the full package. The output of “the truck” has multiple outs including the full mix of all grafix, some only have the mix minus any branding, etc. While the isos get recorded in the truck, they are not pushed out to the broadcast center.

All of the “mixing” as you call it is done in the truck. If you’ve never seen it, it is quite impressive. In one part of the truck is the director and the technical director. The director is the one calling things like “ready camera 1”, “take 1”, etc. the TD is the one on the switcher pushing the actual buttons on the console to make it happen. Next to them is the graphics team prepping all of the stats made available to the TD to key in. In another area is the team of slomo/replay that are taking the feeds from all of the cameras to recorders that allow the operators to pull out the selects and make available for the director/TD to cut to. Typically in the back of the truck is the audio mixer that mixes all of the mics around the event in real time. All of that creates the signal you see on your screen. It leaves the back of the truck and heads out to wherever the broadcaster has better control

"I flew into LHR on Monday" - frequent flyer

"I flew into EGLL on Monday" - pilot

It's the core purpose of the organization. The ticker represents the most fundamental drive of the org. Seems appropriate and honest.

>I think it is easier to have millions of tvs catch airwaves vs millions of point to point https video streams.

Exactly! It was a solved problem.

The Superbowl isn't even the biggest. World Cup finals bring in billions of viewers.

Technically, you forgot capitalization, so it's more keystrokes

But it's "note well", not just "note"

Yeah, I was switching between my phone and desktop to watch the stream and I had a seamless experience on both devices the entire time. I’m not sure why so many people are assuming this was a universal experience.

AAPL is only fewer keystrokes than Apple if you’re in a physical keyboard and hold the shift key, which makes it hardly more convenient. If you use caps lock presumably you’ll press it again.

On a phone, at least in iOS, you have to double tap the shift key.

Not sure if you’re joking, but if not: there’s no practical difference is what is being asked of the reader. Nobody predicates any decision on whether they were asked to note something vs note it well.

It’s pointless jargon.

Not nowadays, more and more remote production for larger and larger events, and it’s coming on rapidly. Directors are increasing sitting in centralised control rooms rather than in a scanner.

BT sport are interesting, spin up graphics, replay, etc in an AWS environment a couple of hours before. I was impressed by their uefa youth league coverage a couple of years ago, and they aren’t slowing down

https://www.limitlessbroadcast.tv/portfolio/uefa-youth-leagu...

https://www.svgeurope.org/blog/headlines/stratospheric-revol...

Obviously not every broadcast, or even most, are remote now, but it’s an ever increasing number.

I don’t know how the US industry works, I suspect the heavy union presence I’ve seen at places like NAB will slow it, but in Europe remote production is increasingly the future.

BigTechMercenaryNews ?

I chose to interpret it charitably and assume OP was saying it's not pushed from venue direct to viewer.

> We use fixed point links and satellite where possible because we own the whole pipe.

Over long distance I get better reliability out of a decent internet provision than in many fixed point to point links, and certainly when comparing at a price point. The downside of the internet is you can't guarantee path separation - even if today you're routing via two different paths, tomorrow the routes might change and you end up with everything going via the same data centre or even same cable.

You probably are a major investor, incidentally, through your pension fund(s) and retirement savings. It is hard to avoid Netflix if you are doing any kind of broad-based index investing.

Is the goal “show the fight” or “use this technology”?

I guarantee the people trying to watch the fight cared more about watching the fight than how the fight was watched.

I used to gather fringe signals for shitty hedge funds, “fintech” from public communities such as HN.

I think I subconsciously adopted it since it made my job easier. Sort of how I use YYYYMMDD format in almost everything from programming to daily communication.

not really. It’s just something I adopted as part of a job I had for fintech 3-4 yrs ago

I don’t do it as some sort of “signaling” for “fintech bros” or anything like that

Apple is also capitalized, so it's still fewer. Unless you hit "Shift" separately for each character in AAPL?