Estimates are difficult for developers and product owners

Tasks that are easiest to estimate are tasks that are predictable, and repetitive. If I ask you how long it'll take to add a new database field, and you've added a new database field 100s of times in the past and each time they take 1 day, your estimate for it is going to be very spot-on.

But in the software world, predictable and repetitive tasks are also the kinds of tasks that are most easily automated, which means the time it takes to perform those tasks should asymptotically approach 0.

But if the predictable tasks take 0 time, how long a project takes will be dominated by the novel, unpredictable parts.

That's why software estimates are very hard to do.

As you say, worthwhile software is usually novel. And to justify our expense, it needs to be valuable. So to decide whether a project is worth doing, we're looking at some sort of estimate of return on investment.

That estimate will also, at least implicitly, have a range. That range is determined by both the I and the R. If you don't have a precise estimate of return, making your estimate of investment more precise doesn't help anything. And I've never seen an estimate of return both precise and accurate; business is even less certain than software.

In my opinion, effort put into careful estimates is almost always better put into early, iterative delivery and product management that maximizes the information gained. Shipping early and often buys much clearer information on both I and R than you can ever get in a conference room.

Of course all of this only matters if running an effective business is more important than managerial soap opera and office politics. Those often require estimates in much the same way they're required from Star Trek's engineers: so the people with main character syndrome have something to dramatically ignore or override to prove their dominance over the NPCs and material reality.

This is an interesting assumption. I’d argue that the overwhelming majority of software is the most boring LoB CRUD apps you can imagine, and not novel at all. Yet, people need to estimate the tasks on these projects as well.

If something is truly boring in software, it gets turned into a library or a tool for non-programmers to use. Our value is always driven by the novelty of the need.

And no, people don't need to estimate the tasks. My dad did LoB apps in the 1970s to the 1990s. E.g., order entry and shop floor management systems for office furniture factories. His approach was to get something basic working, see how it worked for the users, and then iteratively improve things until they'd created enough business advantage and/or cost savings to move on. Exploratory, iterative work like that can at best be done with broad ballpark estimates.

I grant that people want estimates. But that is usually about managerial fear of waste and/or need for control. But I think there are better ways to solve those problems.

[1] e.g., https://en.wikipedia.org/wiki/DBase

I think it's important that our "work culture" allows us to critique why we do certain tasks. When push comes to shove, I guess a manager can say: "Because I say so!", but I also hope those kind of managers are few and far between.