Most active commenters
  • kortilla(4)
  • JCharante(4)
  • (3)
  • bsimpson(3)
  • nicce(3)
  • spike021(3)
  • squeaky-clean(3)
  • colesantiago(3)
  • YZF(3)
  • brokenmachine(3)

←back to thread

492 points storf45 | 148 comments | | HN request time: 1.098s | source | bottom
1. freditup ◴[] No.42154036[source]
I wonder if there will be any long term reputational repercussions for Netflix because of this. Amongst SWEs, Netflix is known for hiring the best people and their streaming service normally seems very solid. Other streaming services have definitely caught up a bit and are much more reliable then in the early days, but my impression still has always been that Netflix is a step above the rest technically.

This sure doesn't help with that impression, and it hasn't just been a momentary glitch but hours of instability. And the Netflix status page saying "Netflix is up! We are not currently experiencing an interruption to our streaming service." doesn't help either...

replies(27): >>42154059 #>>42154082 #>>42154106 #>>42154115 #>>42154122 #>>42154127 #>>42154144 #>>42154158 #>>42154174 #>>42154237 #>>42154240 #>>42154262 #>>42154269 #>>42154313 #>>42154369 #>>42154377 #>>42154390 #>>42154537 #>>42154579 #>>42154690 #>>42154772 #>>42154800 #>>42154855 #>>42154957 #>>42155322 #>>42155792 #>>42155880 #
2. ocdtrekkie ◴[] No.42154059[source]
So the issue is that Netflix gets its performance from colocating caches of movies in ISP datacenters, and a live broadcast doesn't work with that. It's not just about the sheer numbers of viewers, it's that a live model totally undermines their entire infrastructure advantage.

See: https://openconnect.netflix.com/en/

replies(9): >>42154072 #>>42154074 #>>42154076 #>>42154079 #>>42154138 #>>42154193 #>>42154199 #>>42154343 #>>42161993 #
3. pinkmuffinere ◴[] No.42154072[source]
Damn that sucks. I wonder if they could have intentionally streamed it 5 min late? I don’t have all the context around the fight though — maybe a competing service would win if Netflix intentionally induced delay?
replies(1): >>42154793 #
4. ◴[] No.42154074[source]
5. stingraycharles ◴[] No.42154076[source]
Correct, this is not Netflix’ regular cup of tea, and it’s a very different problem to solve. They can probably use their edge caches, but it’s challenging.
replies(1): >>42154146 #
6. sgarland ◴[] No.42154079[source]
I wonder how effective it would be to cache live events with a delay. Write to the tail, read from the head.
replies(2): >>42154189 #>>42154204 #
7. jimbob45 ◴[] No.42154082[source]
Yeah, the funny part is that Hulu, Amazon Prime, and Peacock have all demonstrated the ability to handle an event of this caliber with no issue. Netflix now may never get another opportunity like this again.
replies(3): >>42154117 #>>42154198 #>>42157322 #
8. bsimpson ◴[] No.42154106[source]
I don't spend much time streaming, but I got a glimpse of the Amazon Prime catalog yesterday, and was surprised at how many titles on the front page were movies I'd actually watch. Reminded me of Netflix a dozen years ago.
replies(6): >>42154124 #>>42154142 #>>42154151 #>>42154169 #>>42154329 #>>42154333 #
9. ilaksh ◴[] No.42154115[source]
I think they have to refund the fees for a month to anyone who streamed this fight. That's the only thing that seems fair.

It has been pretty useless. At the moment seems to be working only when running in non-live mode several minutes behind.

So if there are 1 million trying to stream it, that means they would lose $15 million. So.. they might only give a partial refund.

But people should push for an automatic refund instead of a class action.

replies(1): >>42154165 #
10. dyauspitr ◴[] No.42154117[source]
Sure they will. They’ll just set up the next event and outside of some tech folks no one will remember this.
replies(1): >>42154254 #
11. jeromegv ◴[] No.42154122[source]
Not the same demographic but their last large attempt at live was through a Love is blind reunion. It was the same thing, millions of people logging in, epic failure, nothing worked.

They never tried to do a live reunion again. I suppose they should have to get the experience. Because they are hitting the same problems with a much bigger stake event.

replies(3): >>42154156 #>>42154424 #>>42157261 #
12. nicce ◴[] No.42154124[source]
Amazon Prime front page includes currently a lot of ads for movies that you can rent or ”buy”. Are you sure these movies weren’t them?
replies(1): >>42154356 #
13. Spooky23 ◴[] No.42154127[source]
It may vary by ISP. It’s been fine for me.
replies(1): >>42155006 #
14. bushbaba ◴[] No.42154138[source]
Not sure I fully buy that. The “live” stream is rarely “live”. It’s often a highly cached buffer that’s a few mins from latest. Those in isp caches can still help here.
15. LollipopYakuza ◴[] No.42154142[source]
Prime Video has to be the worst of all major streaming services. The video quality is horrible, its crippled with ads (3 not skippable ads for an episode of 45 minutes, lastly), and a lot of interesting titles are behind a "partner paywall".
replies(4): >>42154173 #>>42154191 #>>42154669 #>>42154680 #
16. rvz ◴[] No.42154144[source]
I don't think Netflix is even designed to handle very extreme multi-region live-streaming at scale as evidenced in this event with hundreds of millions simultaneously watching.

YouTube, Twitch, Amazon Prime, Hulu, etc have all demonstrated to stream simultaneously to hundreds of millions live without any issues. This was Netflix's chance to do this and they have largely failed at this.

There are no excuses or juniors to blame this time. Quite the inexperience from the 'senior' engineers at Netflix not being able to handle the scale of live-streaming which they may lose contracts for this given the downtime across the world over this high impact event.

Very embarrassing for a multi-billion dollar publicly traded company.

replies(1): >>42154200 #
17. nicce ◴[] No.42154146{3}[source]
How YouTube does this? Netflix is like drop in the ocean when compared to.
replies(3): >>42154194 #>>42154203 #>>42154344 #
18. stackghost ◴[] No.42154151[source]
> Reminded me of Netflix a dozen years ago.

It's been pretty rough the last few years. So many great films and series, not to mention kids programming, removed to make way for mediocre NeTfLiX oRiGiNaLs and Bollywood trash.

replies(2): >>42154187 #>>42155165 #
19. barkingcat ◴[] No.42154156[source]
yup wanted to say that live stream stuttering has happened before on Netflix - I don't think the reputation is deserved.

From a livestreaming standpoint, netflix is 0/x - for many large events such as love is blind, etc.

From a livestreaming standpoint, look to broadcast news, sports / Olympics broadcasters, etc and you'll see technology, equipment, bandwidth, planning, and professionalism at 1000x of netflix.

Heck, for publicly traded quarterly earnings livestream meetings, they book direct satellite time in addition to fiber to make sure they don't rely only on terrestrial networks which can fail. From a business standpoint, failure during a quarterly meeting stream can mean the destruction of a company (by making shareholders mad that they can't see and vote during the meeting making them push for internal change) - so the stakes are much higher than live entertainment streaming.

Netflix is good at many things, livestreaming is not one of those things.

replies(2): >>42155883 #>>42156514 #
20. 2OEH8eoCRo0 ◴[] No.42154158[source]
Is cable and broadcast better for live TV? No scaling issues. Doesn't matter how many people tune in.
replies(3): >>42154455 #>>42154551 #>>42154571 #
21. SoftTalker ◴[] No.42154165[source]
Is it really that big a deal if you are watching a few minutes behind?

I've watched ball games on streaming networks where I can also hear a local radio broadcast, and the stream is always delayed compared to the radio, sometimes by quite a lot. But you'd never know it if you were just watching the stream.

replies(4): >>42154188 #>>42154202 #>>42154308 #>>42155172 #
22. Spooky23 ◴[] No.42154169[source]
Netflix pivoted to be a platform to waste as much of your time as possible vs entertain.

The Amazon originals are way better imo. They do the dark pattern crap with paid content, as one would expect from Amazon.

replies(1): >>42154375 #
23. unsnap_biceps ◴[] No.42154173{3}[source]
I was watching the rings of power and it started with a "Commercial free experience provided by so and so" with a long ad at the start of the episode, and then a third of the way into the episode, at a critical action part, it broke in the middle of the actor's sentence to a 6 minute ad block.

I exited playback and haven't gone back to finish it. I'll wait for it eventually to make it to a Blu-ray release someday.

24. oaththrowaway ◴[] No.42154174[source]
I used to work for a live streaming platform once. We always joked that VOD (Netflix) was "easy" compared to live.
replies(1): >>42154744 #
25. notatoad ◴[] No.42154188{3}[source]
>Is it really that big a deal if you are watching a few minutes behind?

i don't bet on sports. but from friends who do: yes, it's a really really big deal.

replies(1): >>42164550 #
26. thefreeman ◴[] No.42154189{3}[source]
that’s totally unacceptable for live sports which people are able to bet on
replies(5): >>42154257 #>>42154342 #>>42154423 #>>42154645 #>>42154753 #
27. m463 ◴[] No.42154191{3}[source]
I have prime and my shopping experience is crippled with ads too.

I think it got worse for sellers recently too. If I search for something, like a specific item using its description, sometimes the only result for it shows "sponsored".

It used to show up as sponsored and also unsponsored below.

If this changed, I assume it is bad for the seller. Either they pay for all search results, or their metrics are skewed because all searches were helped by "sponsorship" (and there are no longer unsponsored hits)

28. h4l ◴[] No.42154193[source]
That model still works for streaming. You have a central source stream only to the distributed edge locations, then have clients only stream from their local edge location. Even if one region is overwhelmed, the rest can still work. Load on the central source is bounded.
29. Bilal_io ◴[] No.42154194{4}[source]
Not sure how Netflix does it. But this is not very time sensitive, and I would have delayed the stream by 15 to 30 seconds to cache it and then deliver to everyone.
30. silisili ◴[] No.42154198[source]
AFAIK those three farm out to CDNs with tons of edges who know what they're doing.

I have a feeling Netflix said 'how hard could this be?' and is finding out right now.

replies(1): >>42160984 #
31. _dark_matter_ ◴[] No.42154199[source]
I don't think that live doesn't work with caches. No one watching live would care about a O(s) delay, which is highly amenable to caching at ISPs and streaming changes from there to downstream clients. Offhand I'd say that would support O(ms) delay but no less.
32. gonzo41 ◴[] No.42154200[source]
They must have try to do this on the cheap, thinking they could dynamically scale on the fly for this. Big mistake.
replies(2): >>42154236 #>>42154242 #
33. defrost ◴[] No.42154202{3}[source]
Potentially more so in this brave new world of increased sports betting.

https://en.wikipedia.org/wiki/The_Sting

Offshore combined streaming and betting houses will be cleaning up the rubes.

replies(1): >>42157790 #
34. unsnap_biceps ◴[] No.42154203{4}[source]
My wild assed guess is the differences in the edge nodes.

Netflix's edge nodes are optimized for streaming already encoded videos to end users. They have to transcode some number of formats from the source and send them all to the edge nodes to flow out. It's harder to manage a ton of different streams flowing out to the edge nodes cleanly.

I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients. Only one stream from source to edge to worry about and is much simpler to support and reason about.

But that's just my wild assed guess.

replies(1): >>42154516 #
35. andreimackenzie ◴[] No.42154204{3}[source]
I would be surprised if they don't already do this. The question is how big a buffer to trade off for delay...
36. anakaine ◴[] No.42154236{3}[source]
This is a total supposition without any proof.
replies(1): >>42154709 #
37. noncoml ◴[] No.42154237[source]
> their streaming service normally seems very solid

Not trying to downplay their complexity, but last I heard Netflix is splitting the shows in small data chunks and just serves them as static files.

Live streaming is a different beast

replies(2): >>42154275 #>>42154502 #
38. a_random_name ◴[] No.42154240[source]
If Netflix still interviews on hacker rank puzzles I think this should be a wake up call. Interviewing on irrelevant logic puzzles is no match for systems engineering.
replies(2): >>42154448 #>>42154847 #
39. fragmede ◴[] No.42154242{3}[source]
Their CDN is colo and doesn't run on AWS.
40. CitrusFruits ◴[] No.42154254{3}[source]
I mean, I guarantee you every boxing fan is never going to trust Netflix again for an event like this.
replies(2): >>42154304 #>>42154664 #
41. dullcrisp ◴[] No.42154257{4}[source]
Live sports require microwave relays for high frequency sports bets
42. that_guy_iain ◴[] No.42154262[source]
I don't think it'll be long-term. Most people will forget about this really quickly. It's not like there will be many people saying "Oh, you don't want to sign up for Netflix, the Tyson fight wasn't well streamed" in even 6 months nevermind 10 years.
43. lawgimenez ◴[] No.42154269[source]
There's an upcoming NFL game on Netflix next month. They need to get their shit together.
replies(1): >>42154852 #
44. treflop ◴[] No.42154275[source]
Static files have been pretty much the standard streaming protocols for both VOD and live for the last 15 years ago. Before, it was Adobe Flash (RTMP).

With the way that they are designed, you can even use a regular CDN.

replies(1): >>42154829 #
45. dyauspitr ◴[] No.42154304{4}[source]
What are they going to do? Just not watch the next fight?
46. ilaksh ◴[] No.42154308{3}[source]
The issue is that most people are trying to watch live which is what it's advertised as. And until they figure out that they need to watch X minutes behind, it is unwatchable. Many will not figure that out.

So for the first hour it was just total frustration until I stopped trying to go back to live mode.

47. jpalawaga ◴[] No.42154313[source]
From what I've heard, Netflix has really diluted the culture that people know of from the Patty McCord days.

In particular, they have been revising their compensation structure to issue RSUs, add in a bunch of annoying review process, add in a bunch of leveling and titles, begin hiring down market (e.g. non-sr employees), etc.

In addition to doing this, shuffling headcount, budgets, and title quotas around has in general made the company a lot more bureaucratic.

I think, as streaming matured as a solution space, this (what is equivalent to cost-cutting) was inevitable.

If Netflix was running the same team/culture as it was 10 years ago, I'd like to say that they would have been able to pull of streaming.

replies(1): >>42154954 #
48. ◴[] No.42154329[source]
49. spike021 ◴[] No.42154333[source]
Amazon Prime isn't so great. Lots of for rent/purchase content or content with ads these days. And they end up repeating slots of content in all the rows in their UI, so I end up seeing the same suggestions everywhere rather than much that's new (other than first party productions).

To me they're basically padding their front page.

But honestly that's most of the major streaming platforms these days. I recently cancelled Disney Plus for similar reasons. The only reasons I don't cancel prime or Netflix are because I have family members I split the memberships with to share.

replies(3): >>42154474 #>>42154730 #>>42157111 #
50. rk06 ◴[] No.42154342{4}[source]
Why should they catering to such an audience in first place?

I think this could be one of upsells that Netflix could use.

Premium: get no delay

Normal users: get cache and delay

replies(1): >>42154450 #
51. ethbr1 ◴[] No.42154343[source]
I'm curious if the root cause is more variable than usual latency.

Sample size 1, but...

I saw a ton of buffering and failure on an embedded Netflix app on a TV, including some infinite freezes.

Moved over to laptop, zero buffering.

I assume the web app runs with a lot bigger buffer than whatever is squeezed into the underpowered TV.

replies(1): >>42155094 #
52. spike021 ◴[] No.42154344{4}[source]
In my experience even YouTubeTV has problems sometimes. I'll have the 1080p (and enhanced mode also I think) quality set and still deal with a lot of compression artifacts.
53. joshuakcockrell ◴[] No.42154356{3}[source]
This is my biggest issue with Prime Video. You never know what's included and what costs extra.
replies(2): >>42154409 #>>42157378 #
54. squeaky-clean ◴[] No.42154369[source]
Has Netflix ever live streamed something before? People on reddit are reporting that if you back up the play marker by about 3 minutes the lag goes away. They've got a handle on streaming things when they have a day in advance to encode it into different formats and push it to regional CDNs. But I can't recall them ever live streaming something. Definitely nothing this hyped.
replies(2): >>42154488 #>>42154601 #
55. djbusby ◴[] No.42154375{3}[source]
Every Amazon show looks the same, yellow washed or something; and they should spend more money on costumes - they get beat by low budget cosplay.
replies(1): >>42154462 #
56. Fire-Dragon-DoL ◴[] No.42154377[source]
For me, netflix constantly forget the last episode/spot I was in a TV show. Beyond frustrating
57. geodel ◴[] No.42154390[source]
I think Netflix will have even more sw engineers looking to work there once they notice even for average quality of work they can get paid 3 times more than their current pay.
58. squeaky-clean ◴[] No.42154409{4}[source]
There's also the "FreeVee" items, which have ads regardless of whether you're a prime subscriber or not. And it feels like a lot of their catalog has been transferred over to FreeVee.
59. Brybry ◴[] No.42154423{4}[source]
This is kind of silly because the delay between actual event happening to showing up on OTA TV or cable TV to showing up on satellite TV can already be tens of seconds.
replies(1): >>42155183 #
60. KennyBlanken ◴[] No.42154424[source]
Is anyone surprised? I don't see how their infrastructure can handle this when it was designed for non-realtime precaching of prerecorded content.
61. rvz ◴[] No.42154448[source]
For systems design and engineering, absolutely this. I expected the very highest standards and upmost uptime from Netflix, similar to Google and Amazon.

Tells you the uselessness of their engineering blogs.

62. ctvo ◴[] No.42154450{5}[source]
Or, hear me out here, it's a wild concept, just work.

You know, like every other broadcaster, streaming platform, and company that does live content has been able to do.

Acting like this is a novel, hard problem that needs to be solved and we need to "upsell" it in tiers because Netflix is incompetent and live broadcasting hasn't been around for 80+ years is so fucking stupid.

replies(1): >>42154647 #
63. Valord ◴[] No.42154455[source]
Seems like it
64. nicce ◴[] No.42154462{4}[source]
Fallout was pretty good. Very loyal to the game.
65. chamomeal ◴[] No.42154474{3}[source]
I recently found a lil dvd rental place in my city. It’s a non-profit, they also do archivals and stuff.

It’s pretty much a two-story townhouse packed head to toe with DVDs (lots of blu rays!)

You don’t realize how limited the streaming collection is until you’re back in a movie store, looking through thousands and thousands of movies you would never find otherwise.

Since I found it, I’ve started doing movie night every week with my friends. It’s such an absolute blast. We go to the store, each pick out a random movie that looks good (or bad, in a good way) or just different.

All of a sudden, I love movies again!!

replies(3): >>42154493 #>>42154738 #>>42154783 #
66. ipaddr ◴[] No.42154488[source]
Love is blind reunion which had major problems. Chris Rock comedy which only had a few issues. The Netflix Cup which had issues.
67. spike021 ◴[] No.42154493{4}[source]
That's an excellent option. I think it'd be remiss not to mention local libraries. Of course, your mileage may vary, but the ones I've gone to do seem to have adequate selections. I just don't often make time to go there and browse like I would have at traditional video rental places back in the day.

Heck, mine even have some video games; though from when I've checked they're usually pretty back-reserved.

68. astrange ◴[] No.42154502[source]
That's how streaming (usually) works. The main URL is a playlist of transport stream files and it just downloads them in the background as you go.
69. vitus ◴[] No.42154516{5}[source]
> I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients.

Ha, no, our edge nodes don't have anywhere near enough spare CPU to do transcoding on the fly.

We have our own issues with livestreaming, but our system's developed differently over the past 15 years compared to Netflix's. While they've historically focused on intelligent pre-placement of data (which of course doesn't work for livestreaming), such an approach was never feasible for YT with the sheer size of our catalog (thanks to user-generated content).

Netflix is still new to the space, and there isn't a good substitute for real-world experience for understanding how your systems behave under wildly different traffic patterns. Give them some time.

replies(1): >>42154640 #
70. test6554 ◴[] No.42154537[source]
Yea, it’s a bad look. But I switched to watching some other Netflix video and it seemed fine. Just this event had some early issues. Looks fine now though.
71. betaby ◴[] No.42154551[source]
Interestingly that TV nowadays is delivered through IP.
72. vitus ◴[] No.42154571[source]
It's fundamentally different, for sure.

Most third-party internet-based streaming solutions are overlaid on top of a point-to-point network, while broadcast is one-to-many, and even cable tends to use multicast within the cable provider's network.

You have potentially different problems, e.g. limited bandwidth / spectrum. If, say, there are multiple games going on at the same time, you can only watch whichever feed the broadcaster decides to air. And, of course, regardless of the technology in use, there are matters of acquiring rights for various events. One benefit of internet-based streaming is that one service can acquire the rights and be able to reach everyone, whereas an individual cable provider might only reach its direct subscribers.

replies(1): >>42154740 #
73. rapind ◴[] No.42154579[source]
> ut my impression still has always been that Netflix is a step above the rest technically.

I always assumed youtube was top dog for performance and stability. I can’t remember the last time I had issues with them and don’t they handle basically more traffic than any other video service?

replies(1): >>42154914 #
74. StevePerkins ◴[] No.42154601[source]
Chris Rock comedy special, and the Tom Brady roast. Nothing on this scale, though.
75. kortilla ◴[] No.42154640{6}[source]
It also helps that youtube serves shit tier quality videos more gracefully. Everyone is used to the step down to pixel-world on youtube to the point where they don’t complain much.
replies(2): >>42154713 #>>42169632 #
76. kortilla ◴[] No.42154645{4}[source]
I have bad news for you. This is how it works already for “live” sports
replies(2): >>42155103 #>>42158227 #
77. kortilla ◴[] No.42154647{6}[source]
Every other live platform has a delay of multiple seconds
78. kortilla ◴[] No.42154664{4}[source]
0 people give a shit about boxing fans. It’s not up to them.
79. RaftPeople ◴[] No.42154669{3}[source]
> Prime Video has to be the worst of all major streaming services

I would put Prime Video at 2nd worst. Absolute worst IME is Paramount+.

Edit: worst for streaming quality

80. hughesjj ◴[] No.42154680{3}[source]
It's also super annoying to try to watch on a computer compared to Netflix or YouTube
81. colesantiago ◴[] No.42154709{4}[source]
What more proof do you need other than the fact that streams went down worldwide on a highly anticipated event from a public company?

I wouldn't be surprised if lots of engineers at Netflix are currently now writing up a length post mortem of this.

And this is from the company that created the discipline of chaos engineering for resilience.

It is clear they under invested and took the eye of the ball with this.

This is bad, like very very bad.

replies(1): >>42155091 #
82. Ekaros ◴[] No.42154713{7}[source]
And decent part of these users are on free tier, so they are not paying for it. That alone gives you some level of forgiveness. At least I am not paying anything for this experience.
83. bsimpson ◴[] No.42154730{3}[source]
I've given Netflix a lot more money than I've gotten value out of. I've had an account for ~15y and only really use it for airplanes unless there's a specific thing I'm excited to watch.

I'm in the same boat where as soon as they make it too hard to share, I'll probably cancel it. I think the main reason their sharing crackdown hasn't been a problem so far is that I use it so seldomly, it thinks the "main" address is my parents, which makes it easy for me to pass the "are you traveling" 2FA on my own phone when I do want to watch something.

84. bsimpson ◴[] No.42154738{4}[source]
I was in high school in the early 00s, and going to the movies was such a big part of my life. Now, I never even know what's out.

I suspect life stage is a factor, but it does feel like there are many classes of entertainment (cinema and standup come to mind) that don't resonate like they used to.

replies(1): >>42155830 #
85. Ekaros ◴[] No.42154740{3}[source]
On cable(terrestrial is entirely different) even the bandwidth or spectrum is less limiting for broadcasting multiple games. Hard thing is the other parts of production, like cameras, live-directing and live commentary. Adding new channels is less challenging than actual producing content at expected level there.
86. taeric ◴[] No.42154744[source]
Not really a joke, though? VOD has obvious methods to cheat a bit. Redundancy abounds and you can even network shape for costs. Could probably get even better compression for clear reasons.

Live, not so much. One source that you have to fanout from and absolutely no way to get cheap redundancy. Right?

replies(1): >>42157030 #
87. squeaky-clean ◴[] No.42154753{4}[source]
I don't bet so I have no clue, but why is that? Are people able to place bets in the middle of the match or something? I would have assumed bets get locked in when the fight starts
replies(2): >>42154799 #>>42155179 #
88. kenjackson ◴[] No.42154772[source]
Streamed glitch free for me both on my phone and Xbox. The fight wasn’t so great though, but still a fun event. Jake Paul is a money machine right now.
replies(1): >>42162018 #
89. justinclift ◴[] No.42154783{4}[source]
Heh Heh Heh. Maybe there will be a resurgence of Blockbuster style retail stores... ;)
90. adrr ◴[] No.42154793{3}[source]
they were introducing 5 minute delays on some of the clients. I noticed my ipad was always live and the smart tv had a 5 minute delay but you could fast forward to live.
91. umanwizard ◴[] No.42154799{5}[source]
Idk about traditional sports books but on Polymarket you can certainly continue betting at any time until the market resolves.
replies(1): >>42154960 #
92. ikt ◴[] No.42154800[source]
Based on this I'm wondering whether it was straight up they did not expect it to be this popular?

> Some Cricket graphs of our #Netflix cache for the #PaulVsTyson fight. It has a 40 Gbps connection and it held steady almost 100% saturated the entire time.

https://fosstodon.org/@atoponce/113491103342509883

93. adrr ◴[] No.42154829{3}[source]
You can push these files to all the edges before you release the content which will protect your origin. Livestream all your edge servers are grabbing content from the origin unless you have another tier of regional servers to alleviate load.
replies(1): >>42155164 #
94. VirusNewbie ◴[] No.42154847[source]
I did a round of netflix interviews, didn't get an offer (but passed the technical coding rounds) they absolutely had the best interview process of any company I've interviewed at my entire career.

They do make you code but the questions were 1. Not on hacker rank or leetcode 2. Pratical coding questions that didn't require anything more than basic hashmaps/lists/loops/recursion if you want. Some string parsing, etc.

They were still hard, you had to code a fast, but no tricky algorithms required. It also felt very collaborative, it felt like you were driving pair programming. Highly recommended even though didn't get an offer!

95. caseyohara ◴[] No.42154852[source]
Two games actually, both on Christmas Day. A day when most people are at home or the home of family or friends, and they are both pretty good late-season matchups (Chiefs-Steelers and Ravens-Texans) so I imagine viewership will be high.

If they botch the NFL games, it will surely hurt their reputation.

96. dh2022 ◴[] No.42154855[source]
I think why I will remember about this fight is not the (small) streaming issue I encountered as much as the poor quality of the fight itself. For me that was the reputational loss. Netflix was touting “NFL is coming to Netflix”. This fight did not really make me want to watch that.
replies(1): >>42158620 #
97. normie3000 ◴[] No.42154914[source]
Maybe a client issue, but i've got a low-end smart tv which handles netflix fine, but youtube is unwatchable due to buffering and failed cuts to adverts
replies(1): >>42159224 #
98. neilv ◴[] No.42154954[source]
Were they not able to hire enough top-skilled people? If not, why not?

Or did they have a lot of needs that they decided didn't require top-skilled people?

Or was this a beancounter thing, of someone deciding that the company was paying more money on staffing than they needed to, without understanding it?

replies(2): >>42156619 #>>42156870 #
99. tass ◴[] No.42154957[source]
Was live streaming much of a use case for them before this?

They stream plenty of pre recorded video, often collocated. Live streaming seems like something they aren’t yet good at.

replies(2): >>42155043 #>>42156682 #
100. pests ◴[] No.42154960{6}[source]
They end betting some minutes before the fight ends.

I last saw Tyson at +500 while Jake was around -800 on DraftKings somewhere in the 6th round.

101. graton ◴[] No.42155006[source]
I think you are correct. Ziply Fiber said they were seeing 2.1 times their normal peak [1].

But also people were saying they weren't having any issues streaming on Ziply.

[1] https://www.reddit.com/r/ZiplyFiber/comments/1gsenik/netflix...

102. someothherguyy ◴[] No.42155043[source]
https://www.cnn.com/2023/03/04/entertainment/chris-rock-netf... (2023-03-06)
103. jhugo ◴[] No.42155091{5}[source]
The assumption that it was related to insufficient investment isn’t supported by any evidence. Flawed technical decisions can be made by the most expensive engineers too.
replies(1): >>42157316 #
104. YZF ◴[] No.42155094{3}[source]
Likely these devices use different media formats and/or quality levels. And yes, it's possible one device buffers more than the other. Infinite freezes sounds like some routing issues or bugs.
replies(1): >>42156196 #
105. YZF ◴[] No.42155103{5}[source]
Yep. Having actually worked on this sort of stuff I can confirm.

Your ISP doesn't have enough bandwidth to the Internet (generally speaking) for all users to get their feed directly from a central location. And that central location doesn't have enough bandwidth to serve all users even if the ISP could. That said, the delay can be pretty small, e.g. the first user to hit the cache goes upstream, the others basically get the stream as it comes in to the cache. This doesn't make things worse, it makes them better.

106. treflop ◴[] No.42155164{4}[source]
Sure but that’s why your edge servers do request collapsing. And there are full blown CDN companies that will write an enterprise contract with you that can do this stuff with ease. Akamai is like 25 years old now.

Scale has increased but the techniques were figured out 20 years ago. There is not much left to invent in this space at the current moment so screwing up more than once is a bit unacceptable.

107. JCharante ◴[] No.42155165{3}[source]
is this specifically in India? I never see bollywood stuff in the US but half the catalogue is dubbed/subbed korean dramas
108. JCharante ◴[] No.42155172{3}[source]
well it's live sports, watching live is the big deal. Also people are gambling on the outcomes so watching a few minutes behind is a big deal
109. JCharante ◴[] No.42155179{5}[source]
a match has multiple rounds doesn't it? Seems logical to bet on individual rounds or events that can occur throughout the match.
110. JCharante ◴[] No.42155183{5}[source]
isn't this why people would listen via radio?
111. apexalpha ◴[] No.42155322[source]
>but my impression still has always been that Netflix is a step above the rest technically.

Maybe if we're not counting Youtube as 'streaming', but in my mind no one holds a candle to YT quality in (live)streaming.

112. Yeul ◴[] No.42155792[source]
There's a difference between live broadcasts and serving up content that's sitting on a server I guess?

In my country every time there's a big football match the people who try to watch it on the internet face issues.

113. Yeul ◴[] No.42155830{5}[source]
Back in the day everyone was watching the same thing. The choices for entertainment were limited to whatever was showing in movie theatres, whatever was on TV and whatever record stores were selling.
114. MyFedora ◴[] No.42155880[source]
Netflix won't take a hit here.

Most people pay Netflix to watch movies and tv shows, not sports. If I hadn't checked Hacker News today, I wouldn't even know they streamed sports, let alone that they had issues with it. Even now that I do, it doesn't affect how I see their core offering, which is their library of on-demand content.

Netflix's infrastructure is clearly built for static content, not live events, so it's no shock they aren't as polished in this area. Streaming anything live over the internet is a tough technical challenge compared to traditional cable.

replies(1): >>42155956 #
115. glimshe ◴[] No.42155883{3}[source]
Even some of the old guard can do this. The Olympics worked pretty well (despite the awkward UI), and that was Peacock/NBC.

Perhaps Netflix still needs a dozen more microservices to get this right...

116. oplav ◴[] No.42155956[source]
Netflix is trying to expand into live sports. This event wasn’t a one off thing. There is an NFL game they are streaming at the end of the year.
117. ethbr1 ◴[] No.42156196{4}[source]
When I was watching the behavior on the tv, was wondering if buffering sends some separate, non-business-as-usual requests, and that part of Netflix's delivery architecture was being overloaded.

E.g. "give me this previous chunk" vs "send me the current stream"

replies(1): >>42159593 #
118. emeril ◴[] No.42156514{3}[source]
All valid points though each of those examples seemingly only has a fraction of the viewers of the netflix events, right?
replies(1): >>42158035 #
119. jpalawaga ◴[] No.42156619{3}[source]
Combination of 2 and 3. The business changed. Streaming was more or less a solved problem for Netflix. They needed money for content, not expensive engineers. Ted is co-ceo… you can see where the priority is.
120. fred_is_fred ◴[] No.42156682[source]
If places like Paramount+ can figure it out, Netflix, given their 10+ year head start on streaming and strong engineering culture, should also have been able to. And if you don't like my example, literally every other streaming service has streamed live sports without issue. YT TV, Hulu, Paramount+, Amazon Prime, Peacock, even Apple TV streams live sports.

It may be "new" to them, but they should have been ready.

replies(1): >>42158430 #
121. AtlasBarfed ◴[] No.42156870{3}[source]
My observation is that Netflix is one of those places that brags about how they do so much with so little employees.
replies(1): >>42158386 #
122. oaththrowaway ◴[] No.42157030{3}[source]
Yes, of course but it was still a cope because we never saw more than maybe 3M concurrent viewers at a time
123. 0x0000000 ◴[] No.42157111{3}[source]
> And they end up repeating slots of content in all the rows in their UI, so I end up seeing the same suggestions everywhere rather than much that's new

All of the streaming services do this and I hate it. Netflix is the worst of the bunch, in my experience. I already scrolled past a movie, I don't want to watch it, don't show it to me six more times.

Imagine walking through a Blockbuster where every aisle was the same movies over and over again.

124. tylerchilds ◴[] No.42157261[source]
this is false, the tom brady roast was live streamed

yes, love is blind failed, but was definitely not the most recent attempt. they did some other golf thing too, iirc

replies(1): >>42157274 #
125. anshumankmr ◴[] No.42157274{3}[source]
tom brady is largely a guy popular in the USA whereas Mike Tyson is globally famous. It follows that this fight would attract a larger audience.
replies(1): >>42157541 #
126. colesantiago ◴[] No.42157316{6}[source]
The evidence is that the stream went down.

We will see why it went down and to what extent they underinvested in their post mortem.

replies(1): >>42162896 #
127. anshumankmr ◴[] No.42157322[source]
Chances are Jake might fight Connor McGregor. Sure, Connor is not as famous as Tyson but that will also invite a large number of people to stream it.
128. HDThoreaun ◴[] No.42157378{4}[source]
You can toggle to only show free content. Still get ads but theyre obvious.
129. tylerchilds ◴[] No.42157541{4}[source]
the point i’m making is that the netflix live streaming timeline didn’t go

chris rock -> love is blind -> mike tyson

they have had other, successful executions in between. the comment i was replying to had cherry picked failures and i’m trying to git rebase them onto main.

130. SoftTalker ◴[] No.42157790{4}[source]
Internet streams are not real-time even in the best case. There is always a few seconds of delay, often quite a bit more than that depending on number of hops and link speeds, congestion, etc.
131. barkingcat ◴[] No.42158035{4}[source]
for livestreams, individual events like the Olympics probably has a surge audience of 10x of netflix events.

Netflix events is small potatoes compared to other livestream stalwarts.

Imagine having to stream a cricket match internationally to UK / India / Australia with combined audience that crushes the Superbowl or a football match to all of Europe, or even something like livestreaming F1 racing that has multiple magnitudes of audience than a boxing match and also has 10x the number of cameras (at 8K+ resolution) across a large physical staging arena (the size of the track/course) in realtime, in addition to streaming directly from the cockpit of cars that are racing 200mph++.

Livestream focused outfits do this all day, everyday.

Netflix doesn't even come close to scratching the "beginner" level of these kinds of live events.

It's a matter of competencies. We wouldn't expect Netflix to be able to serve burgers like McDonald's does - Livestreaming is a completely different discipline and it's hubris on Netflix's part to assume just because they're good at sending video across the internet they can competently do livestreaming.

132. jrpelkonen ◴[] No.42158227{5}[source]
Correct. Here are some latency numbers from the last SuperBowl: https://www.phenixrts.com/resource/super-bowl-2024

Even the best latency is dozens of seconds behind live action.

133. dboreham ◴[] No.42158386{4}[source]
Few. Little employees would be...small.
134. tass ◴[] No.42158430{3}[source]
I won’t argue that they shouldn’t have done better, I’m only pointing out that this is fairly different from their usual product. Amazon, YouTube, and Hulu all have a ton of experience with live streaming by now. Apple has live streamed wwdc for several years.

I did expect that Netflix would have appropriately accounted for demand and scale, though, especially given the hype for this particular event.

replies(1): >>42174470 #
135. BLKNSLVR ◴[] No.42158620[source]
I don't care about boxing or UFC or the grade-A douchebags that are the Paul brothers, but I tuned in just because I had the time and a Netflix subscription.

It was actually great that the fight itself was so boring because it justifies never having to spend time / money on that kind of bullshit. It was a farce. A very bright, loud, sparkly, and expensive (for some people) farce.

The value I got from it was the knowledge that missing out on that kind of thing isn't really missing out on anything at all.

136. rapind ◴[] No.42159224{3}[source]
Maybe that’s it. I pay from premium though so don’t have the advert issue (apples to apples).
137. YZF ◴[] No.42159593{5}[source]
Buffering typically just consumes the same live stream until there's enough in the buffer. No difference other than the request rate being potentially higher. At least I can confidently say that for the standard players/video platforms. NetFlix could be doing something different. I'm not sure if they have their own protocols. But I'd be very surprised if the buffering used a completely different mechanism.
138. ◴[] No.42160984{3}[source]
139. tnvmadhav ◴[] No.42161993[source]
they will learn :)
140. tnvmadhav ◴[] No.42162018[source]
same. could be a bandwidth issue at CDN/ISP level at a certain region(s) ?
141. jhugo ◴[] No.42162896{7}[source]
That’s not evidence for the assertion you made.
replies(1): >>42166780 #
142. throwaway2037 ◴[] No.42164550{4}[source]
I have seen this type of comment a few times here. Why does it matter to a bettor if the stream is delayed by 1-5 mins? How would they even know?
replies(1): >>42169610 #
143. colesantiago ◴[] No.42166780{8}[source]
Other potential and future entertainment partners Netflix will be working with e.g. WWE, will certainly see my view as they will be questioning Netflix's capability after that major streaming issue we both saw.

This isn't Netflix's first time they had this live streaming problem.

People will see this as an underinvestment from Netflix's part and they will reconsider going to a different streaming partner.

144. brokenmachine ◴[] No.42169610{5}[source]
I remember a few years ago reading about a scam at the Australian Open Tennis where there were people inside the stadium who were betting on individual points as they happened.

I guess they could bet before the betting streams caught up.

It seems ridiculous to me that you can bet on individual points, but here we are.

145. brokenmachine ◴[] No.42169632{7}[source]
I stream hours of 4k60 from youtube every day for free.

I get maybe 1m total of buffering per week, if that.

Seems uncharitable to complain about that.

replies(1): >>42172216 #
146. bobdvb ◴[] No.42172216{8}[source]
Live streams have different buffering logic to video on demand. Customers watching sports will get very upset if there is a long buffer, but for a VOD playback you don't care how big the buffer is. Segment sizes are short for live and long for VOD because you need to adapt faster and keep buffers small for Live, but longer download segments are better for buffering.
replies(1): >>42177550 #
147. bobdvb ◴[] No.42174470{4}[source]
Even Amazon doesn't have it perfected for Live.

Live sports and VOD movies/TV are very different beasts.

148. brokenmachine ◴[] No.42177550{9}[source]
Sorry, yeah, for some stupid reason I was not thinking about live streams.