But what if I want a faster horse?

For any given thing or category of thing, a tiny minority of the human population will be enthusiasts of that thing, but those enthusiasts will have an outsize effect in determining everyone else's taste for that thing. For example, very few people have any real interest in driving a car at 200 MPH, but Ferraris, Lamborghinis and Porsches are widely understood as desirable cars, because the people who are into cars like those marques.

If you're designing a consumer-oriented web service like Netflix or Spotify or Instagram, you will probably add in some user analytics service, and use the insights from that analysis to inform future development. However, that analysis will aggregate its results over all your users, and won't pick out the enthusiasts, who will shape discourse and public opinion about your service. Consequently, your results will be dominated by people who don't really have an opinion, and just take whatever they're given.

Think about web browsers. The first popular browser was Netscape Navigator; then, Internet Explorer came onto the scene. Mozilla Firefox clawed back a fair chunk of market share, and then Google Chrome came along and ate everyone's lunch. In all of these changes, most of the userbase didn't really care what browser they were using: the change was driven by enthusiasts recommending the latest and greatest to their less-technically-inclined friends and family.

So if you develop your product by following your analytics, you'll inevitably converge on something that just shoves content into the faces of an indiscriminating userbase, because that's what the median user of any given service wants. (This isn't to say that most people are tasteless blobs; I think everyone is a connoisseur of something, it's just that for any given individual, that something probably isn't your product.) But who knows - maybe that really is the most profitable way to run a tech business.

> you will probably add in some user analytics service, and use the insights from that analysis to inform future development. However, that analysis will aggregate its results over all your users, and won't pick out the enthusiasts, who will shape discourse and public opinion about your service. Consequently, your results will be dominated by people who don't really have an opinion, and just take whatever they're given.

This is so spot on. I was a long-time serial entrepreneur who spent a couple decades across three successful startups discovering, shipping and growing new categories of tech products primarily for consumer, prosumer and hobbyists. Then I sold my last startup to a very large F500 silicon valley tech leader and ended up a senior product exec there. While there were a lot of positives like more mature engineering processes, testing and devops as a discipline, the exact issue you describe was a nightmare of product-damaging mistakes I called "analytics abuse." In my startups I valued having increasingly robust analytics over the years. In part because they helped increase my overall understanding of usage but mostly because they provoked good questions to explore. That exploration happened naturally because as the "product guy / founder" I never stopped spending a lot of time with our most passionate, opinionated, thought-leading customers. Over years of iteration I'd learned how to engage deeply and listen carefully to input from these customers. This involved interpreting, filtering and curating the mess of divergent personal preferences and pet feature ideas to tease out the more actionable product signals that could increase broad usage, adoption and passion around our products. I'd then bring those curated signals back to the product teams for evaluation and prioritization.

At BigCo they were diligent about meeting with customers, in fact they had entire processes around it, but their rigorous structures and meeting agendas often got in the way of just directly engaging and actively listening. Worse, the customer meetings the more senior product decision makers actually attended in person were mostly with the highest revenue customers. Junior PMs (and sometimes new grads) were delegated to meeting with the broader base of customers and filing reports. Those reports were then aggregated by ever-helpful program managers into tables of data and, eventually, slides - losing all nuance and any ability to spot an emerging outlier signal and tug on that thread to see where it goes.

I tried to convince everyone that we were missing important customer signals, especially from our smartest, most committed users. Being only one level removed from the CEO and quite credible based on prior success, I was definitely heard and most people agreed there was something being lost but no one could suggest a way to modify what we were doing that could scale across dozens of major products and hundreds of product managers, designers, execs and other stakeholders. In my experience, this general problem is why large companies, even the most well-run, successful ones full of smart people trying their best, end up gradually nerfing the deeper appeal in their own products. Frustratingly, almost every small, single step in that long slide pushes some short-term metric upward but the cumulative effect is the product loses another tiny piece of the soul that made our most evangelistic, thought-leading customers love the product and promote it widely. Ultimately, I ended up constantly arguing we should forego the uplift from some small, easy-to-prove, metric-chasing change to preserve some cumulative whole most people in the org weren't fully convinced even existed. It was exhausting. And there's no fighting the tide of people incentivized on narrow KPIs come bonus season.

I'm sorry to report I never found a solution to this problem, despite my best efforts over several years. I think it's just fundamental. Eventually I just told friends, "It's a genetic problem that's, sadly, endemic to the breed" (the 'breed' being well-run, very large tech companies with the smartest product people HR can hire at sufficient scale). Even if I was anointed CEO, given the size of the product matrix, I could only have personally driven a handful of products. I do think codifying premises and principles from the CEO level can help but it still gets diluted as the number of products, people and processes scales.

Given several mrandish-equivalents, gathered into a side-channel Customer Advocacy org, is there some way to integrate their output without this problematic constantly arguing against metric-chasing?

I'm groping towards something vaguely ombudsman-y, or WW2 production/logistics trouble shooters. Or maybe even pre-Bush41 ARPA Project Managers - term-limited person-with-a-checkbook and few accountability constraints.

If one accepts this role has to be out-of-band, vs poking big hairy blob in hope of creating and maintaining signal channels with particular properties, and grants CEO-adjacent leverage, then it seems a remaining unresolved challenge is integrating the output signals at scale? If so, maybe (jest) CA granted KPI offsets?