Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

Every time a big company screws up, there are two highly informed sets of people who are guaranteed to be lurking, but rarely post, in a thread like this:

1) those directly involved with the incident, or employees of the same company. They have too much to lose by circumventing the PR machine.

2) people at similar companies who operate similar systems with similar scale and risks. Those people know how hard this is and aren’t likely to publicly flog someone doing their same job based on uninformed speculation. They know their own systems are Byzantine and don’t look like what random onlookers think it would look like.

So that leaves the rest, who offer insights based on how stuff works at a small scale, or better yet, pronouncements rooted in “first principles.”

you are so right about that. tho I'm sure that many of the netflix folks are still doing their after action analysis in prep for Dec 25 NFL.

now take this realization and apply it to any news article or forum post you read and think about how uninformed they actually are.

Right? A common complaint by outsiders is that Netflix uses microservices. I'd love to hear exactly how a monolith application is guaranteed to perform better, with details. What is the magic difference that would have ensured the live stream would have been successful?

The only time I worked on a project that had a live television launch, it absolutely tipped over within like 2 minutes, and people on HN and Reddit were making fun of it. And I know how hard everyone worked, and how competent they were, so I sympathize with the people in these cases. While the internet was teeing off with easy jokes, engineers were swarming on a problem that was just not resolving, PMs were pacing up and down the hallway, people were getting yelled at by leadership, etc. It's like taking all the stress and complexity of a product launch and multiplying it by 100. And the thing I'm talking about was just a website, not even a live video stream.

It's not guaranteed, but much fewer points of failure.

I doubt a "microservice" has anything to do with delivering the video frames. There are specific kinds of infrastructure tech that are specifically designed to serve live video to large amounts of clients. If they are in fact using a "microservice" to deliver video frames, then I'd ask them to have their heads examined. Microservices are typically used to do mundane short-lived tasks, not deliver video.

what was the ultimate cause/fix of issues in your case? a database thing?

> people were getting yelled at by leadership

this is where you get up and leave

> It's not guaranteed, but much fewer points of failure.

Can you explain where this is relevant to buffering issues?

Also, you are very wrong regarding failure modes. The larger the service, the more failure modes it has. Moreover, in monoliths if a failure mode can take down/degrade the whole service, all other features are taken down/degraded. Is having a single failure mode that brings down the whole service what you call fewer points of failure?

For an event like this, there already exists an architecture that can handle boundless scale: torrents.

If you code it to utilize high-bandwidth users upload, the service becomes more available as more users are watching -- not less available.

It becomes less expensive with scale, more available, more stable.

The be more specific, if you encode the video in blocks with each new block hash being broadcast across the network, just managing the overhead of the block order, it should be pretty easy to stream video with boundless scale using a DHT.

Could even give high-bandwidth users a credit based upon how much bandwidth they share.

With a network like what Netflix already has, the seed-boxes would guarantee stability. There would be very little delay for realtime streams, I'd imagine 5 seconds top. This sort of architecture would handle planet-scale streams for breakfast on top of the already existing mechanism.

But then again, I don't get paid $500k+ at a large corp to serve planet scale content, so what do I know.

Torrents are awful for live events.

1. Everyone only cares about the most recent "block". By the time a "user" has fully downloaded a block from Netflix's seedbox, the block is stale, so why would any other user choose to download from a peer rather from netflix directly?

2. If all the users would prefer to download from netflix directly rather than a p2p user, then you already have a somewhat centralized solution, and you gain nothing from torrents.

There’s very likely a dedicated service for delivering frames.

That’s service would technically be a “microservice” even if it is a large service.

The protocol for a torrent is that random parts of a file get seeded to random people requesting a file, and that the clients which act as seeds are able to store arbitrary amounts of data to then forward to other clients in the swarm. Do the properties about scaling still hold when it's a bunch of people all requesting real time data which has to be in-order? Do the distributed Rokus, Apple TVs, Fire TVs and other smart TVs all have the headroom in compute and storage to be able to simultaneously decode video and keep old video data in RAM and manage network connections with upload to other TVs in their swarm - and will uploading data to other TVs in the swarm not negatively impact their own download speeds?

Those are the times when you identify who is there to help and who is there to be performative.

I can't, since I don't know Netflix's architecture - I was responding to "I'd love to hear exactly how a monolith application is guaranteed to perform better, with details."

I've noticed this amongst the newer "careerist" sort of software developer who is stumbling into the field for money, as opposed to the obsessive computer geek of yesteryear, who practiced it as a hobby. This character archetype is a transplant, say, less than five years ago from another, often non-technical discipline, and was taught or learned from overly simplistic materials that decry systems programming, or networking, or computer science concepts as unnecessary, impractical skills, reducing everything to writing JavaScript glue code between random NPM packages found on google.

Especially in a time where the gates have come crashing down to pronouncements of, "now anybody can learn to code by just using LLMs," there is a shocking tendency to overly simplify and then pontificate upon what are actually bewilderingly complicated systems wrapped up in interfaces, packages, and layers of abstraction that hide away that underlying complexity.

It reminds me of those quantum woo people, or movies like What the Bleep Do We Know!? where a bunch of quacks with no actual background in quantum physics or science reason forth from drastically oversimplified, mathematics-free models of those theories and into utterly absurd conclusions.

Yes, the properties about scaling do hold even with near-real-time streams. [1]

The problems with using it as part of a distributed service have more to do with asymmetric connections: using all of the limited upload bandwidth causes downloads to slow. Along with firewalls.

But the biggest issue: privacy. If I'm part of the swarm, maybe that means I'm watching it?

[1]: Chainsaw: P2P streaming without trees, https://link.springer.com/chapter/10.1007/11558989_12

If Netflix were working correctly and could handle the load, you'd absolutely be correct.

But it does seem the capacity of a hybrid system of Netflix servers plus P2P would be strictly greater than either alone? It's not an XOR.

And note that in this case of "live" streaming, it still has a few seconds of buffer, which gives a bandwidth-delay product of a few MB. That's plenty to have non-stale blocks and do torrent-style sharing.

I don’t pay my ISP each month to be part of a streaming sites infrastructure. I pay the streaming site each month to use theirs.

You cannot leave us hanging like that. What was the issue?

Insufficient testing

You don’t belong to either group. What does this make you?

What does this have to do with the topic being discussed?

I am one of the ones who complain about their microservices architecture quite a lot.

This comes from both first-hand experience of talking to several of their directors when consulted upon on how to make certain systems of theirs better.

It's not just a matter of guarantees, it's a matter of complexity.

Like right now Google search is dying and there's nothing that they can do to fix it because they have given up control.

The same thing happened with Netflix where they wanted to push too hard to be a tech company and have their tech blogs filled with interesting things.

On the back end they went too deep on the microservices complexity. And on the front end for a long time they suffered with their whole RxJS problem.

So it's not an objective matter of what's better. It's more cultural problem at Netflix. Plus the fact that they want to be associated with "Faang" and yet their product is not really technology based.

Then, instead of people complaining about buffering issues, you'd get people complaining about how the greedy capitalists at Netflix made poor Joe Shmoe use all of his data cap, because they made him upload lots of data to other users and couldn't be bothered to do it themselves.

While that may be the case, the things like this I've experienced have been more along the lines of incompetent management.

In one case I was doing an upgrade on an IPTV distribution network for a cable provider (15+ years ago at this point). This particular segment of subscribers totalled more than 100k accounts. I did validation of the hardware and software rev installed on the routers in question prior to my trip to the data center (2+ hour drive). I informed management that the currently running version on the router wasn't compatible with this hardware rev of card I was upgrading to. I was told that it would in fact work, that we had that same combination of hw/sw running elsewhere. I couldn't find it when I went to go look at other sites. I mentioned it in email prior to leaving I was told to go.

Long story short, the card didn't work, had to back it out. The HA failover didn't work on the downgrade and took down all of those subscribers as the total outage caused a cascading issue with some other gear in this facility. All in all it was during off-peak time of day, but it was a waste of time and customer sat.

Completely agreed. There are also former employees who have very educated opinions about what is likely going on, but between NDAs and whatnot there is only so much they are willing to say. It is frustrating for those in the know, but there are lines they can't or won't cross.

Whenever an HN thread covers subjects where I have direct professional experience I have to bite my tongue while people who have no clue can be as assertive and confidently incorrect as their ego allows them to be.

> they want to be associated with "Faang" and yet their product is not really technology based.

You lost me. Netflix built a massive CDN, a recommendation engine, did dynamic transcoding of video, and a bunch of other things, at scale, quite some years before everyone else. They may have enshittified in the last five years, but I dont see any reason why they dont have a genuinely legitimate claim to being a founder member of the FAANG club.

I have a much harder time believing that companies with AI in their name or domain are doing any kind of AI, by contrast.

You can explain these problems with simple business metrics that technologists like to ignore. Right before the recent Twitter acquisition, the various bits of info that came to the limelight included the "minor detail" that they had more than doubled their headcount and associated expenses, but had not doubled either their revenue or profits. Technology complexity went up, the business went backwards. Thousands of programmers doesn't always translate to more value!

Netflix regularly puts out blog articles proudly proclaiming that they process exabytes of logs per microsecond or whatever it is that their microservices Rube Goldberg machine spits out these days, patting themselves on the back for a heroic job well done.

Meanwhile, I've been able to go on the same rant year after year that they're still unable to publish more than five subtitle languages per region. These are 40 KB files! They had an employee argue with me about this in another forum, saying that the distribution of these files is "harder than I thought".

It's not hard!

They're solving the wrong problems. The problems they're solving are fun for engineers, but pointless for the business or their customers.

From a customer perspective Netflix is either treading water or noticeably getting worse. Their catalog is smaller than it was. They've lost licensing deals for movies and series that I want to watch. The series they're producing themselves are not things I want to watch any more. They removed content ratings, so I can't even pick something that is good without using my phone to look up each title manually!

Microservices solve none of these issues (or make it worse), yet this is all we hear about when Netflix comes up in technology discussions. I've only ever read one article that is actually relevant to their core business of streaming video, which was a blog about using kTLS in BSD to stream directly from the SSD to the NIC and bypassing the CPU. Even that is questionable! They do this to enable HTTPS... which they don't need! They could have just used a cryptographic signature on their static content, which the clients can verify with the same level of assurance as HTTPS. Many other large content distribution networks do this.

It's 100% certain that someone could pretend to be Elon, fire 200-500 staff from the various Netflix microservices teams and then hire just one junior tech to figure out how to distribute subtitles... and that would materially improve customer retention while cutting costs with no downside whatsoever.

You are basically saying, everybody who criticizes Netflix now has no clue.

That’s a bold claim given that people with inside knowledge could post here without disclosing they are insiders.

Is that some kind of No True Scotsman?

Some breaks are just too difficult to predict. For example, I work in ecommerce and we had a page break because the content team pushed too many items into an array, that caused a back-end service to throw errors. Because we were the middle-service, taking from the CMS and making the request to back-end, not sure how we could have seen that issue coming in advance (and no one knew there was a limit).

1. Because Netflix is at capacity? Or because the peer is closer and faster than the original?

And you'll pay less if you become a part

Why is that “very likely”?

I’m genuinely curious about the reasoning behind that statement. It’s very possible that you are using a different set of assumptions or definitions than I am.

You may have belonged to one of those groups in the past, or maybe you will someday. I certainly have. Many of the more seasoned folks on HN have.

Stuff goes wrong, random internet people jump on the opportunity to speculate and say wildly off-the-mark comments, and the engineers trying to keep the ship from sinking have to sit quietly for fear of making the PR backlash worse.

I’m just pointing out that there are Netflix engineers reading all these words.

For every thread like this, there are likely people who are readers but cannot be writers, even though they know a lot. That means the active posters exclude that group, by definition.

These threads often have interesting and insightful comments, so that’s cool.

> who offer insights based on how stuff works at a small scale, or better yet, pronouncements rooted in “first principles.”

And looking through the comments, this is just wrong.

Because it is about people speculating on events that seem connected to their own experience, but in actuality aren’t, because they don’t understand the breadth of the distribution of the abstraction they are discussing.

This happens when your terms are underspecified: someone says “Netflix’s servers are struggling under load” and while people in similar efforts know that basically just equivalent to “something is wrong” and the whole conversation is basically esoteric to most people outside a few specialized teams, these other people jump to conclusions and start having conversations based on their own experience having to do with what is (to them) related (and usually fashionable, because that is how most smaller players figure out how to do things).

Sure. If there’s anything publicly traded companies are known for, it’s passing savings onto their customers instead of their shareholders.

I'm not saying it's easy, but start by assuming that there's a limit and that any request can throw errors? (Proceed accordingly .)

Some people can just let others be wrong and just stay silent, but some people can't help themselves. So if you say something really wrong, like this was caused by Netflix moving to Azure, they should have stayed on AWS! someone will come along to correct you. If you're looking for the right answer, post the wrong one, alongside some provoking statement (Windows is better than Linux because this works there), and you'll get the right answer faster than if you'd asked your question directly.

https://xkcd.com/386/

If NFL decides to keep Netflix for that, that is. The bandwidth for that fight was rookie numbers, and after that fiasco, why would the NFL not break their contract and choose someone with a proven track record doing bigger live events, like the World Cup?

If Netflix is at capacity and you have to wait for a peer, then you have simply reinvented the buffering problem. In other words

1. I exclusively download from a peer and my stream is measurably behind

2. I switch to a peer when Netflix is at capacity and then I have to wait for the peer to download from Netflix, and then for me to download from the peer. This will cause the same buffering issue that Netflix is currently being lambasted for.

This solution doesn’t solve the problem Netflix has

If switching to a peer causes increased buffering (which it will, because you still have to wait for the peer to download from Netflix) then you will still have the original problem Netflix is suffering from.

If the solution to users complaining about buffering is to build a system with more inherent buffering then you are back at square one.

I think it’s might be helpful to look at netlfix’s current system as already a distributed video delivery system in which they control the best seeds. Adding more seeds may help, but if Netflix is underprovisioned from the start you will have users who cannot access the streams

At the scale that Netflix just dealt with? Yeah I honestly think this is a case where less than 5000 people in the world are really qualified to comment.

"Buffering problem" can have very different QOL manifestations, so :

1. You still get a better viewing experience without interruptions. Besides, your "measurably behind" can be an imperceptible fraction of a second?

2. Similar thing - shorter queues - the switch can happen faster due to the extra capacity

So yes, it does solve the practical problem, though not the theoretical one

Because Netflix pays them either way, I would imagine. Breaking a contract on a sure thing to the tune of tens (hundreds?) of millions of dollars for a maybe is a large business risk.

Reputational damage is going to be far more Netflix than the NFL if they totally club it.

That and this fight is going to likely be an order of magnitude more viewers than the Christmas NFL games if the media estimates on viewership were remotely accurate. You’re talking Super Bowl type numbers vs a regular season NFL game. The problems start happening at the margin of capacity most of the time.

The knowledge about that universal practice can be easily acquired.

Yes, and then some idiot with an axe to grind against Logan Paul starts DDoSing people in the Netflix swarm, kicking them out of the livestream. This is always a problem because torrents, by design, are privacy-hostile. That's how the MAFIAA[1] figured out you were torrenting movies in 2004 and how they sent your ISP a takedown notice.

Hell, in the US, this setup might actually be illegal because of the VPPA[0]. The only reason why it's not illegal for the MAFIAA to catch you torrenting is because of a fun legal principle where criminals are not allowed to avail themselves of the law to protect their crimes. (i.e. you can't sue over a drug deal gone wrong)

[0] Video Privacy Protection Act, a privacy law passed which makes it illegal to ask video providers for a list of who watched what, specifically because a reporter went on a fishing expedition with video data.

[1] Music and Film Industry Association of America, a hypothetical merger of the MPAA and RIAA from a 2000s era satire article

Google search is dying because of business reasons, not technical ones. The ads branch is actively cannibalizing search quality to make people perform more searches and view more ads.

"Microservices" have nothing to do with it.

> Right before the recent Twitter acquisition, the various bits of info that came to the limelight included the "minor detail" that they had more than doubled their headcount and associated expenses, but had not doubled either their revenue or profits.

Every tech company massively inflated their headcount during the leadup to the Twitter acquisition because money was free.

I interviewed at Meta in 2021 and asked an EM what he would do if given a magic wand to fix one problem at the company. His response: "I would instantly hire 10,000 more engineers."

Elon famously did the opposite and now revenue is down 80%.

> Some breaks are just too difficult to predict.

Absolutely. I think a great filter for developers is determining how well they understand this. Over-simplification of problems and certainty about one’s ability to build reliable services at scale is a massive red flag to me.

I have to say some of the hardest challenges I’ve encountered were in e-commerce, too.

It’s a lot harder and more interesting than I think many people realize. I learned so much working on those projects.

In one case, the system relied on SQLite and god damn did things go sideways as the company grew its customer base. That was the fastest database migration project I’ve ever been on, haha.

I often think it could have worked today. SQLite has made huge leaps in the areas we were struggling. I’m not sure it would have been a forever solution (the company is massive now), but it would have bought us some much-needed time. It’s funny how that stuff changes. A lot of my takeaways about SQLite 10 years ago don’t apply quite the same anymore. I use it for things now that I never would have back then.

Pornhub has better, more reliable technology than Netflix, yet you don't see their tech blog very often do you?

I say that because, for performance reasons, you’d never want to wait on potentially several hops to stream media and because the act of streaming could very well be a good enough domain boundary.

Those performative people are worse than useless. They take up critical bandwidth and add no real value.

An effective operational culture has methods for removing those people from the conversations that matter. Unfortunately that earns you a reputation for being “cutthroat” or “lacking empathy.”

Both of those are real things, but it’s the C players who claim they are being unfairly treated, when in fact their limelight-seeking behavior is the problem.

If all that sounds harsh, like the kitchen on The Bear, well…that’s kinda how it is sometimes. Not everyone thrives in that environment, and arguably the ones who do are a little “off.”

> I was responding to "I'd love to hear exactly how a monolith application is guaranteed to perform better, with details."

I asked nothing about Netflix. My question was directed at your remark regarding monoliths vs microservices.

Now, can you answer the question?

> You are basically saying, everybody who criticizes Netflix now has no clue.

GP clearly meant some people not everybody. You are the one making bold claims.

Most or all of your replies are to people who hallucinate things you didn’t say. Your patience is inspiring.

All requests expect errors. How a developer handles them... well...

And for limit checking, how often do you write array limit handlers? And if the BE contract doesn't specify? Additionally, it will need as a regression unit test, because who knows when the next developer will remove that limit check.

Money was not "free".

Borrowing costs went to nearly zero. That's not the same thing. You have to repay the money, you just don't have to repay it with interest.

I would have assumed people generally know this, but everybody (and I do mean everybody) talks like they don't know this. I would like to assume that "money is free" is just a shorthand, buuuut... again... these arguments! People like that EM talk like it was literally free money raining from the sky that could be spent (gone!) without it ever having to be repaid.

If you watched any of the long-form interviews Musk gave immediately after the acquisition, he made the point that if he hadn't bailed out Twitter, it had maybe 3 months of runway left before imploding.

Doubling headcount without a clear vision of how that would double revenues is madness. It is doubly so in orgs like Twitter or Netflix where their IT was already over-complicated.

It's too difficult for me to clearly and succinctly explain all the myriad ways in which a sudden inrush of noobs -- outnumbering the old guard -- can royally screw up something that is already at the edge of human capability due to complexity. There is just no way in which it would help matters. I could list the fundamental problems with that notion for hours.

fair.

I have always wondered how do they deliver their content and what goes on behind the scenes and nobody on tech twitter or even youtubers talk about pornhub's infra for some reason. A lot of the innovation in tech has roots in people wanting to see high quality tiddies on the internet.

Monoliths might be better in some cases, microservices in others. Performance can obviously differ based on implementation.

I'm fairly well versed in basic ip networks, but the above sounds like a wordsalad to me.

Even before LLMs were trendy, at the time of covid 19, a lot of people surprisingly became "experts" on the matter of virology and genetics on social networks.

PH has to stream for 20 seconds on average while boxing match is a lot longer :-)

Twitter revive collapsed because of politics and the public removal of moderators, not be side of how many engineers were employed.

Not clear what scale they were attempting, but yes delivering a live stream to 10m+ users on the public internet with a reasonable end to end latency (under 30 seconds glass to viewer) is not a trivial problem, and it’s not something Netflix do a lot.

It’s a very different problem to distributing video on demand which is Netflix’s core business.

Same thing was done a little later by HBO, Disney and a plethora of others, which points to the task not really being uber-difficult.

Use your imagination for just a moment.

The torrent is an example of the system I am describing, not the same system. Torrents cannot work for live streams because the entire content is not hashable yet, so already you have to rethink how it's done. I am talking about adding a p2p layer on top of the existing streaming protocol.

The current streaming model would prioritize broadcasting to high-bandwidth users first. There should be millions of those in a world-scale stream.

Even a fraction of these millions would be enough to reduce Netflix's streaming costs by an order of magnitude. But maybe Netflix isn't interested in saving billions?

With more viewers, the availability of content increases, which reduces load on the centralized servers. This is the property of the system I am talking about, so think backwards from that.

With a livestream, you want the youngest block to take priority. You would use the DHT to manage clients and to manage stale blocks for users catching up.

The youngest block would be broadcast on the p2p network and anyone who is "live" would be prioritizing access to that block.

Torrent clients as they are now handle this case, in reverse; they can prioritize blocks closer the current timestamp to created an uninterrupted stream.

The system I am talking about would likely function at any scale, which is an improvement from Netflix's system, which we know will fail -- because it did.

Speaking as a consumer, Netflix’s solution is objectively better than it’s competitors. It handles network blips better, it’s more responsive to use, it has far fewer random bugs you need to work around.

You can argue whether or not that edge translates into more revenue, but the edge is objectively there.

Based on my experience as a client, they’re not as reliable, Disney in particular. I thought it was a solved problem but it’s not apparently.

I worked on PH infra for a few years -

* The frontend itself runs on bare metal

* A lot of the backend was built out as microservices, running on top of Mesos and then later K8s

* CDN was in-house. The long tail is surprisingly long, but the front page videos were cached globally

The unifying theme behind PH is that they host everything on their own equipment, no AWS/GCP involved.

That’s okay, you probably just haven’t worked with high performance services or micro services before.

Network requests (sometimes called hops) take a significant amount of time. You don’t want your streaming service to take a significant amounts of time.

In microservices land, you generally try making services based on some “domain” (metaphorical, not like a literal domain name) which defines the responsibility of any given service. Defining these domains is more art than science and depends on the business needs and each team.

Video streaming might be one of those domains for Netflix.

Companies weren’t issuing debt to pay for headcount. The reason market interest rates matter is that when interest rates are low, your company stock doesn’t have to have high returns to get investment. When these conditions exist, companies feel safer hiring people to invest in growth instead of saving to provide high shareholder returns.

I highly recommend everyone take a university-level financial instruments course. The math isn’t super hard, and it does a very good job of explaining how rational investors behave.

even for enthusiasts, to go from no programming experience to understanding how Netflix handles video streaming at scale would take more than 5 years.

3) the people supplying 1) and 2) with tools (hard- or software)

We (yep) don't know the exact details, but we do get sent snapshots of full configs and deployments to debug things... we might not see exact load patterns, but it's enough to know. And if course we can't tell due to NDAs.

> Some people can just let others be wrong and just stay silent, but some people can't help themselves.

As one of thoes who cant help themselves; the way you phrase it feels a bit too cynical, I've always interpreted it as people want to help, but don't want to offer something that's wrong. Which is basically how falsifiable science works. It's so much easier to refute the assertion that birds generate lift with tiny backpacks with turboprops attached. Than it is to explain the finer details of avian flight mechanics. I couldn't describe above a superficial level how flapping works, but I can confidently refute the idea of a turboprop backpack. (Everyone knows birds gave up the turboprop design during the great kerosene shortage of 1128)

They want to be associated with FAANG? What does the N stand for?

So you’re saying the investors are happy to see their money set on fire?

Surely they expect at a minimum that their capital investment would make them dividends (increased revenue), and also that the money wasn’t simply set on fire with nothing to show for it and no way to repay it.

If I’m wrong then Twitter - and similar companies - are little better than Ponzi schemes, with investors relying on the money of the greater fool to recover their money.

Tyson was fighting Jake Paul, not Logan Paul. That’s Jake’s brother.

I'm sure 2) can post. But it won't be popular, so you'll need to dig to find it.

Most people are consumers and at the end of the day, their ability to consume a (boring) match was disrupted. If this was PPV (I don't think it is) the paid extra to not get the quality of product they expected. I'm not surprised they dominate the conversation.

Hard to speak "objectively" as a consumer who has their own regional biases and knows none of the sausage underneath.

Maybe you're in a rural area and Netflix scaled gracefully. Maybe you're deep in SF and Netflix simply outspent to give minimal disruption to a population hub. These could both be true but don't speak to what performs better overall.

Sounds like a consumer issue ;)

And nonetheless, it freezes up.

But "reputational damage" doesn't affect profits. Nobody is canceling Netflix because they had issues watching the fight, just like nobody will cancel if the NFL experience sucks on Netflix. They will bitch and moan on Twitter, but it's essentially just talk.

> So you’re saying the investors are happy to see their money set on fire?

Ah, HN, where you try to explain how things work, and you get ignorant sarcasm in return.

> Surely they expect at a minimum that their capital investment would make them dividends (increased revenue), and also that the money wasn’t simply set on fire with nothing to show for it and no way to repay it.

Yes, of course. But when safe investments (e.g., Treasuries) are paying out close to zero, investors are going to tolerate lower returns than they do when Treasuries are paying out 3% or more.

It's basic arithmetic: you take the guaranteed rate, add a risk premium, and that's what investors expect from riskier investments. This is well-covered in the class I recommended.

Also, not every investor thinks in terms of consistent return. A pensioner may have a need for a guaranteed 3% annual return to keep pace with inflation. A VC, on the other hand, is often content to have zero returns for years followed by a 100x payout through an IPO.

I was interviewing a dev candidate some years ago and they were totally lost trying to traverse a tree on the whiteboard. I kept helping them get unblocked, because my philosophy is that anyone can get stuck once, but if I’m supposed decide whether to hire you, I should get the most/best data I can.

Another person was observing the interview, for training purposes, and afterwards said to me: “Do you have kids? You have so much patience!”

If I weren’t retired, I would totally apply to work for you

It depends on the medium and the cost of looking like an idiot. On the Internet where some tosser is going to call you names anyway? Saying dumb shit to nerdsnipe someone else to do hours of research and write an essay on it for you, at the expense of them calling you an idiot, is cheap, and easier than doing all that work yourself. Meanwhile, at work, I'm the one getting nerd sniped into doing a bunch of extra work.

> A VC, on the other hand, is often content to have zero returns for years followed by a 100x payout through an IPO

I know how all this works, but 100x payout is for the small initial investments, not after 10 years of operating at multi-billion-dollar scales.

Small amounts of money are set on fire all of the time, chasing this kind of high-risk return.

Nonetheless, there's an expectation of a return, even if only in aggregate across many small startups.

What I was observing (from the outside, at a distance) was that Twitter was still being run by a startup despite being in an effectively monopoly position already and a "mature" company. Similarly, Amazon could set money on fire while they were the growing underdog. If they doubled their headcount today without doubling either revenue or profits, the idiots responsible for that would be summarily fired.

I get that Silicon Valley and their startup culture does a few things in an unusual way, but that doesn't make US dollars not be US dollars and magically turn into monopoly money that rains from the sky just because interest rates are low.

In short, people with glib answers tend to rely on over simplified models that don’t reflect reality.

If you use Comcast's modem/wifi router, you are part of their service infrastructure. Xfinity WiFi Home Hotspot

People here don't understand basic concepts like risk adjusted returns, flight to quality, Searching for yield etc...

Yeah, try dealing with many frontends with mixed HTTP and HTTPS, it's a nightmare and won't always work. Additionally, you want security on content delivery for revenue protection reasons. The way you've massively over simplified the BSD work shows that you perhaps didn't understand what they did and why hardware offload is a good thing?

Subtitles are also complicated because you have to deal with different media player frameworks on the +40 different players you deal with. Getting those players, which you may not own, to recognise multiple sub tracks can be a PITA.

Things look simple to a junior developer, but those experience in building streaming platforms at scale know there are dragons when you get into the implementation. Sometimes developers and architects do over complicate things, but smart leaders avoid writing code, so its an assumption to say things are being made over complicated.

I am 2, I absolutely will get argued with by people who think they know better.

I'm also not going to criticise my peers because they could recognise me and I might want to work with them one day.

Agree. Hulu, HBO/Max, and Disney Plus each do most of these:

- frequently decide that episodes I've watched are either completely unwatched (with random fully watched eps of the show mixed in).

- seemingly every time I leave at the start of the end-credits, I surely must have intended to come back and watch them.

- rebuild the entire interface (progressively, slowly) when I've left the tab unfocussed for too long. Instead of letting continue where I was, they show it for less than a second, the rebuild the world.

- keep resetting the closed-caption setting to "none", regardless of choosing "always" or "on instant replay"; worse, they sometimes still have the correct setting in the interface, but have disabled captions anyway.

Netflix has only once since they started streaming forgotten playback position or episode completion. They politely suggest when to reload the page (via a tiny footer banner), but even that might not appear for months. They usually know where end-credits really start, and count that as completion. They don't seem to mess with captions.

>"I would instantly hire 10,000 more engineers."

From my experience, this answer usually belies someone who doesn’t fully understand the system and problems of the business. The easy answer when overwhelmed is “we need more people.” To use a manufacturing analogy, you can cover up a lot of quality issues with increased throughout, but it makes for an awfully inefficient system.

Forced assoc.

Honestly, I've seen my fair share of 7 layer SOAP stacks. Not sure if any of your unsubstatiated handwaving is making any sense.

Tell me specifics.

> you perhaps didn't understand what they did

I read and understood their entire technical whitepaper. I get the what, I'm just saying that the why might not make as much sense as you might assume.

> +40 different players you deal with

They own the clients. They wrote the apps themselves. This is Netflix code reading data from Netflix servers. Even if there are third-party clients (wat!?), that doesn't explain why none of Netflix's home-grown clients support more than 5 subtitle languages.

> Getting those players, which you may not own, to recognise multiple sub tracks can be a PITA.

This is a core part of the service, which everyone else has figured out. Apple TV for example has dozens of subtitle languages.[1]

With all due respect: Read what you just wrote. You're saying that an organisation that has the engineering prowess to stream at 200 Gbps per edge box and also handles terabytes of diagnostic log ingestion per hour can't somehow engineer the distribution of 40 KB text files!?

I can't even begin to outline the myriad ways in which these excuses are patent nonsense.

These are children playing with the fun toys, totally ignoring like... 1/3rd of the viewing experience. As far as the users are concerned, there's nothing else of consequence other than the video, audio, and text that they see on the screen.

"Nah, don't worry about the last one, that only affects non-English speakers or the deaf, we only care about DEI for internal hires, not customers."

[1] Just to clarify: I'm asking for there to be an option to select one language at a time from all available languages, not showing multiple languages at once, which is a tiny bit harder. But... apparently not that hard, because I have two different free, open-source video players on my PC that can do this so I can have my spouse get "full" subtitles in a foreign language while I see the "auto" English subtitles pop up in a different colour when appropriate. With Netflix I have to keep toggling between her language and my language every time some foreign non-English thing is said. Netflix is worth $362B, apparently, but hasn't figured out something put together by poor Eastern European hobbyists in their spare time.

See, you're confused because you think that the media player is owned by Netflix.

The browser gives you a certain level of control on computers, although you have to deal with the oddities of Safari, but when you go to smart TVs it's the wild west. Netflix does provide their tested framework to TV vendors but it's still not easy, because media playback often requires hardware acceleration, but the rendering framework isn't standard.

Developing for set-top boxes, multiple generations of phones, and smart TVs comes with all sorts of oddities. You think it's easy because you haven't done it.

Yes, it's on by default, but you can turn this off if you want to. https://www.xfinity.com/support/articles/disable-xfinity-wif...

Shoulda used Varnish.