When did estimates turn into deadlines?

If you're hiring architects and engineers to design and build your home, you might already have a pretty good idea of the home you want. The people you've hired provide estimates on cost and timing based on solidly known quantities. They've put in a basement like that before. They've worked with your weird materials. Their vendors report on material status daily.

Software development is not surrounded by this sort of infrastructure or codification. My discovery process establishes these lines of communication, and I have no idea when I'll uncover a black box or when one will be imposed upon me.

I guess I just don't buy that software is full of unknowns whereas construction is not.

Contractors have to plan for surprises as well. The thing is they've done enough similar work to understand the risks and account for them in budget and timelines.

I think a lot of software engineers, possibly because of the classical world they inhabit, are reluctant to look at things as probabilistic. Your estimate can take into account unknowns you just need to estimate the likeliness of encountering certain snags and the penalty they will impose.

You're framing the problem in terms of mathematical expected value (probability and penalty/reward), but the business environment in which software operates is fundamentally complex (see below).

There is a spectrum of complexity: simple, complicated, complex. These can be framed in terms of ergodicity and you can search for Barry O'Reilly residuality theory if you want to go down this rabbit hole.

In a simple system we can easily predict the future states based on knowledge of past states. In a complicated system, we can also predict future states based on past, but it requires expert knowledge, though it's still fundamentally able to be understood (e.g. SpaceX rockets). These are both ergodic systems. Complex systems are non-ergodic.

Construction is a complicated system that exists within a complicated environment.

Software is a complicated system that exists within a complex environment.

Complex environments can be wrangled along three dimensions: constraining the environment until we can treat it as "only complicated", evolutionary survivorship via random stress and remediation, and avoiding commoditization.

Construction benefits from all three. Environments are constrained to enable the things we build to operate (cars work mostly on paved roads). There is a history of evolutionary survivorship spanning millennia. And construction is less easily commoditized (people typically do the same thing they did yesterday, and repeat for decades). All of this contributes to the ability to a better ballpark estimate.

Software primarily only attempts to constrain the computing environment in which the software runs: If you build your app to run in a container that can tolerate getting yanked and reincarnated elsewhere, you're golden, i.e. cloud computing is an example of constraining a complex environment until it can be treated as "only complicated". But the business environment remains complex and largely unconstrained. We attempt to constrain it via "give us your requirements", but that's more of an anchoring or negotiating technique than actually addressing the complex business environment. Software doesn't have millennia of evolutionary battle testing. And software is more easily commoditized, meaning if you do the same thing you did yesterday on repeat, that turns into a library or an app, so fundamentally you're always being pushed into novel territory, which therefore is less battle tested by evolutionary survivorship. All of this contributes to less clear estimates in software.