Estimates are difficult for developers and product owners

The solution you described is basically agile, and that definitely includes estimates and deadlines.

It’s Agile philosophically, and how it should be.

But that is rarely how it works. In the dozens of different projects across ten or twelve companies I’ve had insight into, “doing Agile” is analogous with “we have a scrum master, hold stand ups, and schedule iterations” while the simple reality is “Agilefall.”

Agreed, but the parent poster said that estimates shouldn't be done at all, which is not a legitimate argument to make in any scenario.

I missed that part of the manifesto

I, and others, don't agree with the blanket statement that "no estimates" is not a legitimate argument in any scenario. Can you expand on why you think there isn't a single case where estimates don't add value? Similarly, is there anything specifically in that post's claims that you think was incorrect, leading to their false conclusion?

There are agile methods that forgo estimates and deadlines though

This is what "agile" is: https://agilemanifesto.org/

More specific methodologies that say they are agile may use concepts like estimates (story points or time or whatever), but even with Scrum I've never run into a Scrum-imposed "deadline". In Scrum the sprint ends, yes, but sprints often end without hitting all the sprint goals and that, in conjunction with whatever you were able to deliver, just informs your backlog for the next sprint.

Real "hard" deadlines are usually imposed by the business stakeholders. But with agile methods the thing they try to do most of all isn't manage deadlines, but maximize pace at which you can understand and solve a relevant business problem. That can often be just as well done by iteratively shipping and adjusting at high velocity, but without a lot of time spent on estimates or calendar management.

I can say with some confidence, having been involved in the movement since before the term "Agile" was coined, that it requires neither.

I grant that both of those are common, but that's because the median "Agile" implementation quickly devolved into mini-Waterfall with more hip names.

I have had many successful projects where we spent approximately zero time on estimates. The fact that a successful approach is culturally seen as illegitimate to even talk about is a great example of why I wrote that last paragraph.

Whenever there are constraints (money, time, resources) there are going to be estimates and prioritisation.

You might be speaking a little more broadly than I am interpreting.

Right? For those who want to check, the four values: https://agilemanifesto.org/

And the 12 principles: https://agilemanifesto.org/principles.html

But to my dismay, the Agile world quickly got colonized by certification schemes and consultants targeting large companies, so it rapidly turned into something that a lot of the early people were very disappointed with. I wrote about the dynamic some years back: https://williampietri.com/writing/2011/agiles-second-chasm-a...

the fact that there was a “manifesto” is always been the funniest shit to me (been hacking since the ‘90’s…)

On the contrary, constraints often mean you don't need formal estimates. (I'll come back to prioritization in a sec.)

Startups are a great example. When you raise your first chunk of money, the size of that isn't really driven by a carefully considered, detailed plan with engineering hours estimated per task. What you get is basically determined what's currently fashionable among angels and small-end VCs, plus who's doing your fundraising. (If you're Jeffery Katzenberg and Meg Whitman, you can raise $1bn. [1] https://en.wikipedia.org/wiki/Quibi But the rest of us have to make do with what we can get.)

So at that point you have a strong constraint (whatever you raised) and some relatively clear goal. As I said, cost isn't nearly as relevant as ROI, and nobody can put real numbers on the R in a startup. At that point you have two choices.

One is just to build to whatever the CEO (or some set of HiPPOs wants). Then you launch and find out whether or not you're fucked. The other is to take something akin to the Lean Startup approach, where you iteratively chase your goal, testing product and marketing hypotheses by shipping early and often.

In that later context, are people making intuitive ROI judgments? Absolutely. Every thing you try has people doing what you could casually call estimating. But does that require an estimation practice, where engineers carefully examine the work and produce numbers? Not at all. Again, I've done it many times. Especially in a startup context, the effort required for estimation is much better put into maximizing learning per unit of spending.

And how do you do that? Relentless prioritization. I was once part of a team that was so good at it that they launched with no login system. Initial users just typed their names in text fields. They wanted proper auth, and eventually they built it, but for demonstrating traction and raising money there were higher priorities. It worked out for them; they built up to have millions of users and were eventually acquired for tens of millions. On very little investor money.

Being great at prioritization makes estimation way less necessary. The units of work get small enough that the law of large numbers is on your side. And the amount of learning gained from the things released change both the R and I numbers frequently enough that formal estimates don't have a long shelf life.

So I get what you're saying in theory, but I'm telling you in practice it's different.

[1] https://en.wikipedia.org/wiki/Quibi

Wait, your first example is "We raised X dollars" which is a literal estimate of the worth of the company?

I think you are well missing the point - everything you put into your rebuttal is about estimates - in time, money, or resources

"Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."

"Working software is the primary measure of progress."

"At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."

Okay, a scenario where you're building a hobby project alone and you don't care if or when it gets finished would be one where estimates aren't needed.

There is no scenario where it's appropriate or necessary when developing software professionally or even as a side project where others are expecting you to complete work at some point.

One of the many misconceptions in the original comment in this thread is that "worthwhile software is usually novel", which is not the case without a very specific and novel definition of worthwhile that I don't believe was intended.

How do you do either of the following without spending any time at all on estimates?

"Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."

"At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."

If I was your colleague, had invested in your company, or was considering giving you money to deliver some product and you refused to tell me when you think it'll be done, it would be the last time any of the above ever happened.

You wouldn't accept that from any service you were attempting to hire, so I don't know why it would be acceptable for software development.

Yes, people keep linking to the agile manifesto as if it's some sort of amulet protecting software developers from any sort of accountability or responsibility for their work product in a professional setting.

It seems like you acknowledge some amount of estimating is needed and I agree that there is an overemphasis on estimation in many places, but I'll ask you the same thing I asked others, which is:

How do you do either of the following without spending any time at all on estimates?

"Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."

"At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."

If you're saying there's some sort of wisps-and-moonbeams notion of estimation in everything that we do, sure. I'm not going to argue with that. One world, brah.

What I am talking about here, though, is a practice of software estimation where programmers produce detailed numbers on the amount of time requested work will take. Which is certainly the common meaning of estimating around here, and also what the original article is about.

"Brah"

I mean, if this is upsetting you, take a minute and go for a walk.

> You might be speaking a little more broadly than I am interpreting.

(Or I could be interpreting it more broadly)

In either case I don't think your response is apt

I feel like I've already answered the how question elsewhere. But the short version is a product-driven kanban-ish approach with frequent releases, small units of work, and engaged product management focused on actual real-world goals. This approach originated 25 years ago in Kent Beck's "Extreme Programming". Which did start with estimates, but teams doing well at it quickly realized (via said regular reflection) that they could do without it and still deliver well.

That some random person on the internet who wouldn't have hired me anyhow will now not hire me is a blow I'll have to learn to live with. But I have literal decades of stakeholders happy working this way. One way that happens is that they pick a date and then we built what fits between now and then. So they get the date, but no promises about what will get delivered.

In practice people like this better because beyond a certain very coarse level, estimates are about feelings of engagement and control. What this approach gives them instead is not just feelings, but actual engagement and control. What they get from me is not some unreliable bullshit date, but a commitment to deliver useful things early and often, so that we can together discover something better than their initial visions.

And yes, of course I accept this from people who do work for me. My side hustle is starting a pinball museum. When volunteers take on repairing a just-arrived machine, I never ask them for estimates. I ask them to focus on doing it right. They discover problems, figure out how to fix them, and try it out to see if there are more problems. The work takes as long as it takes. Or I hired a friend to design the logo. I never asked for a date. We iterated on it, discovering an number of things along the way, including through user tests. It took a lot longer than I would have guessed, and that's great, because it turned out better than I could have hoped.

I understand that this is unfamiliar to people, and apparently it's quite upsetting. But I swear this works. There are more things, Horatio.

None of which requires estimates. And the bit about working software as the primary measure of progress is specifically targeted against the estimate-driven culture of the time, where people would treat GANTT charts, "percentage complete" numbers, etc, as meaningful measures of progress.

I addressed the rest elsewhere, but done well a lack of estimates makes people more accountable. If I am shipping something every week (or as is common for my teams, more often), stakeholders can directly evaluate progress. There's no snowing them with paperwork and claims of progress against semi-fictional documents. They see what they see, they try it out, they watch people use it.

The reality of use is what we in software are ultimately accountable to, and that's what I am suggesting people optimize for. Closing that feedback loop early and often builds stakeholder trust such that they stop asking for elaborate fantasy plans and estimates.

I'm not upset. But as far as I can tell you've wandered off into some sort of philosophical space when I'm speaking of practicalities, without apparent recognition of the transition. I thought that was a bit ridiculous, something from the "wow dude have you ever really looked at your hand" school of conversation, so I made a joke. Apparently the joke didn't land for you. Alas.

It's not possible to do what you're describing (kanban/XP) for any commercial application or open source project that is intended to be taken remotely seriously without some amount of estimating involved.

I saw elsewhere that you're defining estimating as a very detailed and involved process, which the author of the article did not, the person I originally claimed estimates are impossible did not, and I did not.

I agree that's not necessary in most cases, if not all, to do the level of estimating you described.

And you'll note that I didn't include "people who are doing me a favor" in the list of individuals I'd insist on an estimate from.

You don't sound like you're one of these people, but I personally feel that software developers who act like they're performing special incantations over their keyboard and can't be expected to answer to anyone about their deliverables do us all a disservice, though maybe I should just be happy that they allow me to provide an alternative that is much easier to work with and results in additional business for me.

You replied to me in like 10 different places, nearly all of which are in responses to posts that weren't directed at you, so I'm trying not to fragment this discussion too much.

I will ask this here: If you are shipping code to production on a weekly basis, is that not a schedule, also known as a deadline for delivery?

If you expect to ship code to production every week, how do you know whether there will be something to ship without doing any estimation of the effort and time required?

No, shipping code to production on a weekly basis is not a deadline. A deadline is a time by which a task must be completed. A task is something like "fix bug 3831" or "allow users to log in with OAuth2". "Ship code" is not, in any useful sense, a task.

Such "timeboxed iterations" can indeed result in "shipping" a null update. Unless you have a time-consuming QA gate to pass, that's not very likely, especially on a team containing several people, but it can happen. You don't know that you will have "something" to ship.

Typically we try to break changes down into shippable tasks that can be done in under a day, so the expected number of tasks completed by a four-programmer team in a week is on the order of 30, or 15 if you're pairing. For this to fall all the way to 0, everybody has to be spending their time on things that could not be thus broken down. It's pretty unlikely to happen by chance. But sometimes a difficult bug or two really is the thing to focus on.

In XP, estimates are used for prioritizing which tasks to work on and which tasks to break down into smaller tasks. The "product owner" is supposed to choose the tasks that have the most user value for the estimated cost. But those estimates aren't commitments in any sense; they're guesses. Sometimes tasks take more time than estimated; other times, they take less. This is the reason for the shift to estimating in "story points": to prevent the estimates from being interpreted as referring to a period of time.

If someone in your organization is interpreting estimates as commitments, this can have a corrosive effect on the accuracy of those estimates, because estimators respond by padding their estimates, in inconsistent ways. Often this destroys the usefulness of the whole iteration planning process, because the project owner no longer knows which tasks are the easiest ones and thus worth doing even if the benefit is marginal. Organizations can recover from this pathology in a variety of ways, but often they don't. Eliminating estimation is one response that sometimes works.

If software isn't novel, that means some other, existing software does the same thing just as well in the same way on the same platform. So, unless it's a hobby project you're building alone, why don't you just use the existing software?

I think that writing software that isn't novel fails to be worthwhile by a perfectly ordinary, mainstream definition of "worthwhile".

Research is subject to constraints of money, time, and resources, but is not normally estimated in the sense that software industry people would use the term.

Yes, yet estimates are still made. The author of the article didn't use some highly formal definition of estimation, didn't imply one, and seems to be focused on devops (not software development) as a practitioner.

Estimates are difficult, and in unhealthy environments are weaponized against developers. That doesn't mean they're unnecessary or impossible.

So you would consider a CRUD app with some basic business rules to be novel? Basically meaning that any software that requires any development effort is novel?

That's a completely valid definition of worthwhile software, but to claim it's impossible to create an estimate to complete said development is absurd.

It is not a schedule, it's a standard. One I normally try to exceed. We ship when things are ready, which for my current team is ~2-3x/week, but in the past I've had teams that were faster.

We know that there will be things to ship because we try to break the work down into small units of deliverable value by focusing on seeking the highest-value things to do. Large requests are typically composed of a bunch of things of varying value, so we split them until we find things that advance the needs of the user, the customer, or the business. One that's often not intuitive to people is the business need for getting information about what's truly valuable to some part of the audience. So we'll often ship a small thing for a particular audience and see how they react and what actually gets used. (You can save an amazing amount of time by not building the things nobody would end up using.)

Sometimes we can't figure out how to break down something smaller enough that we have something to release right away. Or sometimes a chunk of work surprises us and it drags out. We avoid that, because compared to working on smaller things, it's much less comfortable for both developers and business stakeholders. But if it happens, it happens. We try to learn from it for the next time.

Regarding deadlines, we sometimes have them. Broadly, efforts break down into two categories: driven by date or driven by need. For the former, releasing early and often means we adjust scope to hit the date. For the latter, scope is primary and they get stuff when they get it. Either way because the business side sees steady improvement and has fine-grained control over what gets shipped, they feel in control.

This can be a learning experience for business stakeholders used to waterfall-ish, plan-driven approaches. But I have never once had somebody successfully work this way and want to go back. I have, however, had some product managers get thrown back into document-driven development and tell me how much they missed working like we did.

Yes, this sounds very familiar to me. I started with dates and went to story points used to estimate dates. Then as we turned up the release cadence, we eventually dropped estimating altogether, even in points, because it wasn't really helping anything.

That doesn't mean we refuse to solve the problems that estimates are used to solve. E.g., "Can we have something by the big conference," or "Can we keep our customers happy." We just solve them other ways.

And totally agreed about the corrosive effect of treating estimates as commitments. It's such a negative-sum game, but people play it all the time.

I am always fascinated when people tell me it is impossible to do something I have done. That plenty of people have done. For decades. But I shouldn't be surprised. Kuhn was not kidding around.

And yes, I strongly believe in accountability to business stakeholders, customers, and colleagues. I just think the better way is not paper fantasies, but demonstrated realities. Focusing on working software, customer collaboration, and responding to change, one might say. But people have said it before and it didn't make much difference.

You aren't doing what you're doing without performing what is commonly known as estimating, you just avoid using what seem like triggering words to you and the one other person who is almost spamming responses to my comments with the same information.

See "standard" vs. "schedule", and your very specific and formal definition of an estimate that is almost certainly not what anyone else has been using during this discussion and is not used in practice in most software development shops.

Kudos to you for delivering working features on a consistent enough basis that you've earned enough goodwill that people basically leave you to your internal processes and trust that you'll come through for them. I believe that's necessary to have a healthy working environment where you don't end up with what you, I, and every other software developer on earth is trying to avoid, which is an acrimonious relationship with the people with the money where they dictate what, where, when, and how we do our job.

But to claim that there is literally no estimating or scheduling taking place as you perform software development is just not true. You can post your disagreement on every single comment I've made on this topic if you want, your existing comments already speak for themselves on the matter.

I already said this to your fellow interlocutor who is also responding to nearly every comment of mine with the same thought process, but I'll say it here as well in different terms:

The product owners, customers, salespeople, supervisors, peers, etc. you interact with as part of the software development process on any project outside of a personal hobby don't care about your semantic games.

If functionality is needed in an application, and they ask you to implement it, and you agree, there is no real-world scenario where they just say "Cool, I'll sit idly by while you work at this until you declare it ready, and then and only then will I let anyone else know about it or take any action on its supposed existence," and repeat that for every piece of functionality you implement in perpetuity.

And if you keep failing to deliver required functionality over time, no one is going to accept your arguments that: "Oh sorry, our weekly deliveries to production aren't a deadline, it's a timeboxed iteration", "Oh that estimate wasn't a commitment to do anything, we work on our own schedule", and so on.

Yes, the relationship between developers and "other stakeholders" can turn toxic, but in most organizations the developers don't have much power, probably due to repeated attempts to play the games you've laid out above. The way to combat that is to be reliable and professional so your team has the authority to stand their ground on the difficulty of a given task, not effectively refuse to participate in what is a completely reasonable conversation about the relationship between your work and the objectives of the organization.

You just keep saying things are absurd or obvious but not putting anything behind it.

I hope this isn't a semantics game where things like "1 - 6 months" counts as an estimate in this context.

The point way back up this thread was accurate timelines for complicated, novel work have large error bars but those error bars aren't as bad as the equivalent error bars on estimating whatever "return" it is being pitted against.

I wouldn't consider something like "1-6 months" as a valid estimate, as that would indicate there is too much uncertainty and it needs to be broken down into subtasks that can be estimated with much less variance.

I've written what is probably several pages now in response to two individuals who are redefining terms in order to play the exact semantic games you mentioned, but in order to claim no estimation of any sort needs to be done. We seem to be done talking past each other now that I explicitly pointed out their usage of non-standard terms and my suspicions of why (having also unfortunately lived through software development managed by Gantt chart and other unpleasant experiences where someone who had no idea what they were managing was in control of a project), which is fine with me.

Feel free to describe your experience in practice when working in an organization where software developers answer to no one but themselves and are never asked for any justification for their progress or any projections of when they will be finished (both of which would require estimation to provide).

If you are able to tell stakeholders something like you'll be done in 1-6 months or provide no insight at all into when your tasking will be done, do no tracking of progress internally, and perform no collaboration around the completion of synchronous tasks within your team, I'll acknowledge no estimation is taking place during that process.

Research is estimated, sometimes those estimates are hilariously bad (Computer vision is easy, a summer research for a student should be enough), but more often than not it's "We expect that this research will take someone doing a Ph.D approximately 3 - 5 years to do"

The entire premise of a project is "Look at this, with the intent to find X, and, if it's not possible, break it down so that we can create more projects to work toward that goal" which is an estimate, or a breakdown into sub projects that also come with estimates.

I think that the replies I am getting are demonstrating why developers have estimates used against them - people forget that they are estimates, and they also forget that when new information comes to hand that invalidates that estimate a completely new one may need to be created to take into account the new data.

If developers (or anyone giving estimates) discovers that the initial estimate was based on faulty information then they need to push that information back to whomever they are reporting it to (Team Lead, Product Owner, Manager, customer, angel investor...). The receiver of that information then needs to decide on how to react according to the changes.

Yes, agile is a reaction to spreadsheet driven development and some very dumb ways of tracking progress towards completion and managing work in general.

In my experience, people don't forget they're estimates, they just want to force developers to meet whatever they agreed to that's most convenient for management.

If you want to fight back against that, my experience has been that giving terrible estimates or refusing to give them at all will not result in more autonomy or authority.

> How do you do either of the following without spending any time at all on estimates?

> "Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."

This is 'just' bum-standard continuous delivery (which is where most organizations should be heading). You pull the next todo from the backlog, start working on it. If it takes more than a day to commit something, you split the task into something smaller.

You don't need to estimate ahead of time at all as long as the task is small enough, all you need is to be able to put the near-term backlog of work into a good priority order of business value.

If the high-value task was small it doesn't prevent you from doing more work, because the next unit of work to do is the same either way (the next item on the backlog).

If the high-value task was too big, it can cause you to take a pause to reflect on whether you scoped the task properly and if it is still high-value, but an estimate wouldn't have saved you from it because if you'd truly understood the work ahead of time you wouldn't be pausing to reflect. An estimate, had you performed it, would not have changed the priority.

But this Kanban-style process can be performed without estimates at all, and organizations that work to setup an appropriate context for this will find that they get faster delivery of value than trying to shoehorn delivery into prior estimates instead. But there are people who work faster with the fire of a deadline under their tail so I can't say it's unilaterally better.

> "At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."

If it's hard to do the work as a team, you should be able to tell it was hard retrospectively, with or without having done an estimate ahead of time.

You might say that failing to hit your prior schedule estimates would be a good topic to discuss at a retrospective session, but I would tell you that this is a self-licking ice cream cone. If your customers are happy despite missing internal schedule estimates you're in a good spot, and if your customers are unhappy even though you're "hitting schedule projections" you're in a bad spot.

There's a lot more productive discussions to be done when the team reflects on how things are going, and they typically relate to identifying and addressing obstacles to the continuous flow of value from the product team to the end users.

> If you want to fight back against that, my experience has been that giving terrible estimates or refusing to give them at all will not result in more autonomy or authority.

In my experience giving terrible estimates or refusing to give them at all is the least bad course of action. It wastes less of your time than any realistic alternative, it does no noticeable damage to the business or your own position, and the people who want to paint you as just trying to avoid accountability were going to find a way to do that anyway.

As an outsider, I understand what wpietri argues, but I fail to understand yours --- even though I've read other branches of the discussion. Do you argue that wpietri's method still estimates tasks but the estimation is always around two weeks?