But what if I want a faster horse?

For any given thing or category of thing, a tiny minority of the human population will be enthusiasts of that thing, but those enthusiasts will have an outsize effect in determining everyone else's taste for that thing. For example, very few people have any real interest in driving a car at 200 MPH, but Ferraris, Lamborghinis and Porsches are widely understood as desirable cars, because the people who are into cars like those marques.

If you're designing a consumer-oriented web service like Netflix or Spotify or Instagram, you will probably add in some user analytics service, and use the insights from that analysis to inform future development. However, that analysis will aggregate its results over all your users, and won't pick out the enthusiasts, who will shape discourse and public opinion about your service. Consequently, your results will be dominated by people who don't really have an opinion, and just take whatever they're given.

Think about web browsers. The first popular browser was Netscape Navigator; then, Internet Explorer came onto the scene. Mozilla Firefox clawed back a fair chunk of market share, and then Google Chrome came along and ate everyone's lunch. In all of these changes, most of the userbase didn't really care what browser they were using: the change was driven by enthusiasts recommending the latest and greatest to their less-technically-inclined friends and family.

So if you develop your product by following your analytics, you'll inevitably converge on something that just shoves content into the faces of an indiscriminating userbase, because that's what the median user of any given service wants. (This isn't to say that most people are tasteless blobs; I think everyone is a connoisseur of something, it's just that for any given individual, that something probably isn't your product.) But who knows - maybe that really is the most profitable way to run a tech business.

You’re giving it way too much of a positive spend. None of the companies are using analytics to increase the desirability for the majority of users.

They are doing it to increase “engagement” and so more people will stay on their site longer.

Why else wouldn’t Netflix show the “continue watching” row first instead of forcing you to scroll past algorithmic generated crap?

It is the same reason that Google went from describing success as people getting off their site faster and going to one of the “ten blue links” to the shit show it is today.

What's the difference between that which optimized for what you call "engagement" and what the average user wants?

Presumably the best thing for Netflix is to have a happy userbase, so why do you assume it wouldn't optimize for that?