Most active commenters
  • kortilla(3)
  • YZF(3)

←back to thread

492 points storf45 | 39 comments | | HN request time: 1.052s | source | bottom
Show context
freditup ◴[] No.42154036[source]
I wonder if there will be any long term reputational repercussions for Netflix because of this. Amongst SWEs, Netflix is known for hiring the best people and their streaming service normally seems very solid. Other streaming services have definitely caught up a bit and are much more reliable then in the early days, but my impression still has always been that Netflix is a step above the rest technically.

This sure doesn't help with that impression, and it hasn't just been a momentary glitch but hours of instability. And the Netflix status page saying "Netflix is up! We are not currently experiencing an interruption to our streaming service." doesn't help either...

replies(27): >>42154059 #>>42154082 #>>42154106 #>>42154115 #>>42154122 #>>42154127 #>>42154144 #>>42154158 #>>42154174 #>>42154237 #>>42154240 #>>42154262 #>>42154269 #>>42154313 #>>42154369 #>>42154377 #>>42154390 #>>42154537 #>>42154579 #>>42154690 #>>42154772 #>>42154800 #>>42154855 #>>42154957 #>>42155322 #>>42155792 #>>42155880 #
1. ocdtrekkie ◴[] No.42154059[source]
So the issue is that Netflix gets its performance from colocating caches of movies in ISP datacenters, and a live broadcast doesn't work with that. It's not just about the sheer numbers of viewers, it's that a live model totally undermines their entire infrastructure advantage.

See: https://openconnect.netflix.com/en/

replies(9): >>42154072 #>>42154074 #>>42154076 #>>42154079 #>>42154138 #>>42154193 #>>42154199 #>>42154343 #>>42161993 #
2. pinkmuffinere ◴[] No.42154072[source]
Damn that sucks. I wonder if they could have intentionally streamed it 5 min late? I don’t have all the context around the fight though — maybe a competing service would win if Netflix intentionally induced delay?
replies(1): >>42154793 #
3. ◴[] No.42154074[source]
4. stingraycharles ◴[] No.42154076[source]
Correct, this is not Netflix’ regular cup of tea, and it’s a very different problem to solve. They can probably use their edge caches, but it’s challenging.
replies(1): >>42154146 #
5. sgarland ◴[] No.42154079[source]
I wonder how effective it would be to cache live events with a delay. Write to the tail, read from the head.
replies(2): >>42154189 #>>42154204 #
6. bushbaba ◴[] No.42154138[source]
Not sure I fully buy that. The “live” stream is rarely “live”. It’s often a highly cached buffer that’s a few mins from latest. Those in isp caches can still help here.
7. nicce ◴[] No.42154146[source]
How YouTube does this? Netflix is like drop in the ocean when compared to.
replies(3): >>42154194 #>>42154203 #>>42154344 #
8. thefreeman ◴[] No.42154189[source]
that’s totally unacceptable for live sports which people are able to bet on
replies(5): >>42154257 #>>42154342 #>>42154423 #>>42154645 #>>42154753 #
9. h4l ◴[] No.42154193[source]
That model still works for streaming. You have a central source stream only to the distributed edge locations, then have clients only stream from their local edge location. Even if one region is overwhelmed, the rest can still work. Load on the central source is bounded.
10. Bilal_io ◴[] No.42154194{3}[source]
Not sure how Netflix does it. But this is not very time sensitive, and I would have delayed the stream by 15 to 30 seconds to cache it and then deliver to everyone.
11. _dark_matter_ ◴[] No.42154199[source]
I don't think that live doesn't work with caches. No one watching live would care about a O(s) delay, which is highly amenable to caching at ISPs and streaming changes from there to downstream clients. Offhand I'd say that would support O(ms) delay but no less.
12. unsnap_biceps ◴[] No.42154203{3}[source]
My wild assed guess is the differences in the edge nodes.

Netflix's edge nodes are optimized for streaming already encoded videos to end users. They have to transcode some number of formats from the source and send them all to the edge nodes to flow out. It's harder to manage a ton of different streams flowing out to the edge nodes cleanly.

I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients. Only one stream from source to edge to worry about and is much simpler to support and reason about.

But that's just my wild assed guess.

replies(1): >>42154516 #
13. andreimackenzie ◴[] No.42154204[source]
I would be surprised if they don't already do this. The question is how big a buffer to trade off for delay...
14. dullcrisp ◴[] No.42154257{3}[source]
Live sports require microwave relays for high frequency sports bets
15. rk06 ◴[] No.42154342{3}[source]
Why should they catering to such an audience in first place?

I think this could be one of upsells that Netflix could use.

Premium: get no delay

Normal users: get cache and delay

replies(1): >>42154450 #
16. ethbr1 ◴[] No.42154343[source]
I'm curious if the root cause is more variable than usual latency.

Sample size 1, but...

I saw a ton of buffering and failure on an embedded Netflix app on a TV, including some infinite freezes.

Moved over to laptop, zero buffering.

I assume the web app runs with a lot bigger buffer than whatever is squeezed into the underpowered TV.

replies(1): >>42155094 #
17. spike021 ◴[] No.42154344{3}[source]
In my experience even YouTubeTV has problems sometimes. I'll have the 1080p (and enhanced mode also I think) quality set and still deal with a lot of compression artifacts.
18. Brybry ◴[] No.42154423{3}[source]
This is kind of silly because the delay between actual event happening to showing up on OTA TV or cable TV to showing up on satellite TV can already be tens of seconds.
replies(1): >>42155183 #
19. ctvo ◴[] No.42154450{4}[source]
Or, hear me out here, it's a wild concept, just work.

You know, like every other broadcaster, streaming platform, and company that does live content has been able to do.

Acting like this is a novel, hard problem that needs to be solved and we need to "upsell" it in tiers because Netflix is incompetent and live broadcasting hasn't been around for 80+ years is so fucking stupid.

replies(1): >>42154647 #
20. vitus ◴[] No.42154516{4}[source]
> I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients.

Ha, no, our edge nodes don't have anywhere near enough spare CPU to do transcoding on the fly.

We have our own issues with livestreaming, but our system's developed differently over the past 15 years compared to Netflix's. While they've historically focused on intelligent pre-placement of data (which of course doesn't work for livestreaming), such an approach was never feasible for YT with the sheer size of our catalog (thanks to user-generated content).

Netflix is still new to the space, and there isn't a good substitute for real-world experience for understanding how your systems behave under wildly different traffic patterns. Give them some time.

replies(1): >>42154640 #
21. kortilla ◴[] No.42154640{5}[source]
It also helps that youtube serves shit tier quality videos more gracefully. Everyone is used to the step down to pixel-world on youtube to the point where they don’t complain much.
replies(2): >>42154713 #>>42169632 #
22. kortilla ◴[] No.42154645{3}[source]
I have bad news for you. This is how it works already for “live” sports
replies(2): >>42155103 #>>42158227 #
23. kortilla ◴[] No.42154647{5}[source]
Every other live platform has a delay of multiple seconds
24. Ekaros ◴[] No.42154713{6}[source]
And decent part of these users are on free tier, so they are not paying for it. That alone gives you some level of forgiveness. At least I am not paying anything for this experience.
25. squeaky-clean ◴[] No.42154753{3}[source]
I don't bet so I have no clue, but why is that? Are people able to place bets in the middle of the match or something? I would have assumed bets get locked in when the fight starts
replies(2): >>42154799 #>>42155179 #
26. adrr ◴[] No.42154793[source]
they were introducing 5 minute delays on some of the clients. I noticed my ipad was always live and the smart tv had a 5 minute delay but you could fast forward to live.
27. umanwizard ◴[] No.42154799{4}[source]
Idk about traditional sports books but on Polymarket you can certainly continue betting at any time until the market resolves.
replies(1): >>42154960 #
28. pests ◴[] No.42154960{5}[source]
They end betting some minutes before the fight ends.

I last saw Tyson at +500 while Jake was around -800 on DraftKings somewhere in the 6th round.

29. YZF ◴[] No.42155094[source]
Likely these devices use different media formats and/or quality levels. And yes, it's possible one device buffers more than the other. Infinite freezes sounds like some routing issues or bugs.
replies(1): >>42156196 #
30. YZF ◴[] No.42155103{4}[source]
Yep. Having actually worked on this sort of stuff I can confirm.

Your ISP doesn't have enough bandwidth to the Internet (generally speaking) for all users to get their feed directly from a central location. And that central location doesn't have enough bandwidth to serve all users even if the ISP could. That said, the delay can be pretty small, e.g. the first user to hit the cache goes upstream, the others basically get the stream as it comes in to the cache. This doesn't make things worse, it makes them better.

31. JCharante ◴[] No.42155179{4}[source]
a match has multiple rounds doesn't it? Seems logical to bet on individual rounds or events that can occur throughout the match.
32. JCharante ◴[] No.42155183{4}[source]
isn't this why people would listen via radio?
33. ethbr1 ◴[] No.42156196{3}[source]
When I was watching the behavior on the tv, was wondering if buffering sends some separate, non-business-as-usual requests, and that part of Netflix's delivery architecture was being overloaded.

E.g. "give me this previous chunk" vs "send me the current stream"

replies(1): >>42159593 #
34. jrpelkonen ◴[] No.42158227{4}[source]
Correct. Here are some latency numbers from the last SuperBowl: https://www.phenixrts.com/resource/super-bowl-2024

Even the best latency is dozens of seconds behind live action.

35. YZF ◴[] No.42159593{4}[source]
Buffering typically just consumes the same live stream until there's enough in the buffer. No difference other than the request rate being potentially higher. At least I can confidently say that for the standard players/video platforms. NetFlix could be doing something different. I'm not sure if they have their own protocols. But I'd be very surprised if the buffering used a completely different mechanism.
36. tnvmadhav ◴[] No.42161993[source]
they will learn :)
37. brokenmachine ◴[] No.42169632{6}[source]
I stream hours of 4k60 from youtube every day for free.

I get maybe 1m total of buffering per week, if that.

Seems uncharitable to complain about that.

replies(1): >>42172216 #
38. bobdvb ◴[] No.42172216{7}[source]
Live streams have different buffering logic to video on demand. Customers watching sports will get very upset if there is a long buffer, but for a VOD playback you don't care how big the buffer is. Segment sizes are short for live and long for VOD because you need to adapt faster and keep buffers small for Live, but longer download segments are better for buffering.
replies(1): >>42177550 #
39. brokenmachine ◴[] No.42177550{8}[source]
Sorry, yeah, for some stupid reason I was not thinking about live streams.