Estimates are difficult for developers and product owners

The unique thing about estimates in software engineering is that if you do it right, projects should be impossible to estimate!

Tasks that are easiest to estimate are tasks that are predictable, and repetitive. If I ask you how long it'll take to add a new database field, and you've added a new database field 100s of times in the past and each time they take 1 day, your estimate for it is going to be very spot-on.

But in the software world, predictable and repetitive tasks are also the kinds of tasks that are most easily automated, which means the time it takes to perform those tasks should asymptotically approach 0.

But if the predictable tasks take 0 time, how long a project takes will be dominated by the novel, unpredictable parts.

That's why software estimates are very hard to do.

And I'd add that the need for them is a sign they aren't worth doing.

As you say, worthwhile software is usually novel. And to justify our expense, it needs to be valuable. So to decide whether a project is worth doing, we're looking at some sort of estimate of return on investment.

That estimate will also, at least implicitly, have a range. That range is determined by both the I and the R. If you don't have a precise estimate of return, making your estimate of investment more precise doesn't help anything. And I've never seen an estimate of return both precise and accurate; business is even less certain than software.

In my opinion, effort put into careful estimates is almost always better put into early, iterative delivery and product management that maximizes the information gained. Shipping early and often buys much clearer information on both I and R than you can ever get in a conference room.

Of course all of this only matters if running an effective business is more important than managerial soap opera and office politics. Those often require estimates in much the same way they're required from Star Trek's engineers: so the people with main character syndrome have something to dramatically ignore or override to prove their dominance over the NPCs and material reality.

The solution you described is basically agile, and that definitely includes estimates and deadlines.

There are agile methods that forgo estimates and deadlines though

This is what "agile" is: https://agilemanifesto.org/

More specific methodologies that say they are agile may use concepts like estimates (story points or time or whatever), but even with Scrum I've never run into a Scrum-imposed "deadline". In Scrum the sprint ends, yes, but sprints often end without hitting all the sprint goals and that, in conjunction with whatever you were able to deliver, just informs your backlog for the next sprint.

Real "hard" deadlines are usually imposed by the business stakeholders. But with agile methods the thing they try to do most of all isn't manage deadlines, but maximize pace at which you can understand and solve a relevant business problem. That can often be just as well done by iteratively shipping and adjusting at high velocity, but without a lot of time spent on estimates or calendar management.

Yes, people keep linking to the agile manifesto as if it's some sort of amulet protecting software developers from any sort of accountability or responsibility for their work product in a professional setting.

It seems like you acknowledge some amount of estimating is needed and I agree that there is an overemphasis on estimation in many places, but I'll ask you the same thing I asked others, which is:

How do you do either of the following without spending any time at all on estimates?

"Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."

"At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."

I addressed the rest elsewhere, but done well a lack of estimates makes people more accountable. If I am shipping something every week (or as is common for my teams, more often), stakeholders can directly evaluate progress. There's no snowing them with paperwork and claims of progress against semi-fictional documents. They see what they see, they try it out, they watch people use it.

The reality of use is what we in software are ultimately accountable to, and that's what I am suggesting people optimize for. Closing that feedback loop early and often builds stakeholder trust such that they stop asking for elaborate fantasy plans and estimates.

You replied to me in like 10 different places, nearly all of which are in responses to posts that weren't directed at you, so I'm trying not to fragment this discussion too much.

I will ask this here: If you are shipping code to production on a weekly basis, is that not a schedule, also known as a deadline for delivery?

If you expect to ship code to production every week, how do you know whether there will be something to ship without doing any estimation of the effort and time required?