When did estimates turn into deadlines?

So when those times have occurred I've (we've more accurately) adopted what I refer to the "deer in the headlights" response to just about anything non-trivial. "Hoo boy, that could be doozy. I think someone on the team needs to take an hour or so and figure out what this is really going to take." Then you'll get asked to "ballpark it" because that's what managers do, and they get a number that makes them rise up in their chair, and yes, that is the number they remember. And then you do your hour of due diligence, and try your best not to actually give any other number than the ballpark at any time, and then you get it done "ahead of time" and look good.

Now, I've had good managers who totally didn't need this strategy, and I loved 'em to death. But for the other numbnuts who can't be bothered to learn their career skills, they get the whites of my eyes.

Also, just made meetings a lot more fun.

Maybe this is a good strategy for dealing with people who aren’t going to judge you for delivering slowly, or for managers who don’t know what the fuck is going on. For managers who do, they will see right through this.

https://youtu.be/8xRqXYsksFg?si=qlG18BB2nNUmt3ub

For what it's worth I've seen pretty much the opposite. I don't know about competent vs. incompetent engineers. But when it comes to experience, I've seen the inexperienced ones giving super low estimates and the experienced people giving larger estimates.

I worked at a place where this management insanity was endemic, which lead to everyone padding all estimates with enough contingency to account for that. Which lad to the design team, and the front-end team, and the backend team, and the QA team, all padding out their estimates by 150 or 200% - to avoid the blame storms they'd seen for missing "deadlines".

Then the Project managers added those all ups and added 150 - 200%. Then the account managers and sales teams added 150 - 200% to the estimated costs before adding margins and setting prices.

Which ended up in literally around 1 million dollars a month to maintain a website which could _easily_ have been handled by a full time team of 8 or 10 decent web and full stack devs. Hell, apart from the 24x7 support requirement, I reckon I know a few great Rails or Django devs who could have done all the work on their own, perhaps with a part time of contracted graphic designer.

That all lasted a handful of years, until the client worked out what was going on, and my company management flew the whole thing into the mountain, with ~100 people losing their jobs and their owed entitlements (I was out about $26K that day.)

The punishment for a commitment culture is inflated estimates.

I think a manager who doesn't know the difference between and estimate and a deadline is one who "[doesn't] know what the fuck is going on," and that's the kind of manager the GP uses this strategy with.

is the essence of this thread

That sounded insane to me… nearly two decades later, with plenty of remote freelance and full time onsite team experience under my belt… and I fully agree. It’s always going to take significantly longer, and if you pretend it’s not, it’s going to come down on your head, like it or not. Always better to underpromise and overdeliver than the other way around.

Giving an "if I had nothing else going on" estimate can be a big trap to fall into - they will only see the number and judge your performance based on that. This dovetails into the problem that untracked but still important work being thankless in low trust environments - not all work can ever be tracked, or else the time to track that work would take as long as doing the work. Examples: literally any emotional labor, time to monitor, time to train, time to document when its not explicitly required, time to solve little problems.

In the environment where none of this counts because its not quantifiable, everyone with knowledge makes themselves into a silo in order to protect perceptions of their performance, and everyone else suffers. I'll go even a little further to say that companies that attempt to have no untracked work are by nature far more sociopathic - thus far there's basically no consequences for sociopathic organizations but I hope one day there will be.

The manager need to know how to make the estimate to know is bullshit.

If the manager knows how to make the estimate and what it represents, there is no need to inflate it for them.

In my experience (been hacking since the '90's before it was cool) great developers are great at estimating things. And these are not outliers, all except 1 great developer I've had pleasure of working with over these years has never been "off" on estimates by any statistically significant margin. but you say anything like that here on HN and it is heresy.

My general opinion is that developers LOVE making everyone believe that software development is somehow "special" from many other "industries" for the lack of a better word and that we simply cannot give you accurate estimates that you can use to make "deadlines" (or better said project plans). and yet most developers (including ones raising hell here and downvoting any comment contrary to "popular belief") are basically doing sht that's been done million times before, CRUD here, form there, notification there, event there etc... It is not like we are all going "oh sht, I wonder how long it'll take to create a sign-up form."

I think we have (so far) been successful at running this "scam" whereby "we just can't accurately estimate it" because of course it is super advantageous to us. and WFH has made this even worse in my opinion - given that we can't provide "accurate estimates" now we can simply pad them (who dare to question this, it is just an estimate, right? can't hold me to the estimate...) and then chill with our wifes and dogs when sh*t is done 6 weeks earlier :)

And the only cure is instead building a company that's tolerant of mistakes while still aspiring to excellence.

The one I've worked at which got the closest had a corporate culture that failures were atrributable to processes, while successes were atrributable to individuals/teams.

Of course that had its own negative side effects, but on the whole it made the company a lot more honest with itself. And consequently got better work out of everyone.

Usually there are just too many unknowns that higher estimate is justified to avoid having to explain why you didn't make it by certain deadline. The estimates I give are not median or average that I expected the task to complete, they are so that I can be 95% sure it's possible to do it and then some.

We could say, always say it will take longer than you think?

Though by this principle, it seems that "overestimates" are likely to be actually accurate?

Joel Spolsky wrote about his time estimation software which recorded the actual time required for completion, and then calculated for each person a factor by which their estimates were off, and this factor was consistent enough that it could be reliably used as a multiplier to improve estimation accuracy.

> Most estimators get the scale wrong but the relative estimates right. Everything takes longer than expected, because the estimate didn’t account for bug fixing, committee meetings, coffee breaks, and that crazy boss who interrupts all the time. This common estimator has very consistent velocities, but they’re below 1.0. For example, {0.6, 0.5, 0.6, 0.6, 0.5, 0.6, 0.7, 0.6}

https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...

You cannot rely 100% on any estimates either, and all you are doing by demanding estimates is creating stress and making people less productive. The meta work imposed by that in itself will make a project take more time, as everyone will be padding their estimates.

Volatility is how much the sprint changes. Sure you can pull one 5 pt ticket out and add in a 3 point and 2 point, but if you do that 12 times in a two week sprint, we will not finish the sprint even if total capacity stays under 40 points.

I would snapshot the sprint each day, so each day I could see how many tickets got removed/added. The end result being I could show my manager, look, when volatility is low, we almost always finish the sprint. When the volatility is high, we don't, it doesn't matter if we are over/under velocity because we don't have the time to properly plan and get clarity on asks. Have our product team think more than two weeks out and we'll deliver. That worked to a degree.

And Joel sidesteps the unknown-unknowns problem in that piece, by discussing boiling down tasks to <1 day chunks.

But what if you need to build a prototype before you sufficiently understand the project and options to decide on an approach? Where does that time get estimated?

The more projects I work on, the bigger of a fan of spiral development [0] I become.

Because, at root, there are 2 independent variables that drive project scheduling -- remaining work and remaining risk.

This estimation problem would drastically simplify if it allowed for "high confidence, 30 days" and "low confidence, 5 days" estimates.

And critically, that could drive different development behavior! E.g. prototype out that unknown feature where most of the remaining technical risk is.

Trying to instead boil that down to an un-risk-quantified number produces all the weird behaviors we see discussed elsewhere in the comments.

[0] https://en.m.wikipedia.org/wiki/Spiral_model

I have the same anecdotal experience with a possible explanation:

Inexperienced engineers often don't see the greater picture or the kind of edge cases that will probably need to be handled ahead of time. I've often had the following type of conversation:

Engineer: "I think that would be a day's work"

Me: "This will need to interact with team X's package. Have you accounted for time spent interacting with them?"

Engineer: "Oh no, I guess two days then"

Me: "Will this account for edge case 1 and 2?"

Engineer: "Ah yes, I guess it would be three days then"

Me: "Testing?"

Engineer: "Maybe let's say a week?"

On the other hand experienced devs might have their judgement clouded by past events:

"Last time we did something with X it blew out by 3 months" - Ignoring the fact that X is now a solved issue

My comment was not really geared towards chaotic organizations but general sense (especially in myriad threads on this forum) that SWEs think that somehow our profession is so “special” that we cannot possible know how long something will take. I simply do not accept that and personally believe that more likely vast majority of SWEs simply suck at their job. there are roughly 4.5 million of us, how many are really good at their jobs (big part of which is estimation)? probably like 0.4% :)

As for software development being special, I really hope that what I've described above is like other engineering disciplines, and we're not special. I don't want to be special, I want to be an engineer, like those who work with aircraft or bridges and what not. I feel like in those fields the concept of estimation is a little more respected. But I'm probably wrong. :^)

I'll mention I've been a professional software developer since the early 90's, not that experience equals veracity. But I've had good success using the system above, and even though the bad managers to good managers was pretty even during that time, the company experienced outstanding success during my tenure (20 years!). In the end, bad managers never last. Good managers, who take reasonable estimates to their superiors, succeeded, where managers who brought "It'll be done July 1st" got doubted because their superiors know it really doesn't work that way.

Negative side effects were about what you'd imagine. Some low performers unjustly shielded themselves. Safeguards were overbuilt as proof "something" was changed to prevent a failure repeat. Executive promotion criteria could get squirrelly. Etc.

But on the whole, I think the individual/team productivity boost and agility created by honesty was a huge net win.

Isn't the entire point of a sprint that, once planning at the start of the sprint is over, you can't change what's in it by reprioritizing? All of product's reprioritizing should be in the backlog, not the sprint, and only affect what the next sprint is going to be, not the current one.

This is software though, if X has actually been done before then it doesn't need to be done again. It is already done.

Task X clearly had the potential to blow out by 3 months, and they are now working on task Y that is similar to X. It is a reasonable position to assume that there are other as-yet-unknown issues that might cause it to blow out by 3 months until someone has demonstrated that all the unknown unknowns are also resolved by doing it quickly. That is just basic evidence based planning.

Instead of giving a single fixed estimate, give one with error bars. "3 months, plus minus 4 weeks". Most engineers know their estimates have error bars, but have somehow been bludgeoned into forgetting to mention them.

It's also helpful from the management side - the size of the error bars makes it immediately clear how confident folks are in the estimate. It allows reasoning about risk. It allows things like "OK, currently we have 30% error bars either way - what are the biggest contributors to that? Can we knock one or two of those out when we spend a few days investigating?"

It's beyond me why we, as a supposed engineering profession, are unable to talk about risk, probabilities, and confidence intervals. And that isn't just on managers.

Things take longer but if you over-estimate the project won’t come in significantly any earlier.

I worked at two places that gave huge bonuses when deadlines were met (based on “estimates”) and wouldn’t you know it sh*t always got done on time and people got paid.

Because management quits listening after hearing "3 months," and bad management heard "3 months minus three weeks" and goes "okay 2 months it is".

I dread to imagine the number of bike shedding meetings and planning poker it takes to change a 5 to a 3 or 7.

"provided they control most of the dependencies" is a pretty important caveat by the way. many times I've seen people get the rug pulled out from under them by partner teams at the last second. it doesn't matter how clever you are or how hard you work. if you depend on something owned by a team far away in the org chart, they can always blow up your project with little consequence.

Definitely did not seen this. Under performers are underestimating or just do wild random guesses. Under performance is most likely to be in the form of "making small estimate, try to make it technically, but then it has about millions of problems".

Big estimates require courage and confidence - under performers usually do not have either. They are too scared to estimate high.

1. Ask the dev team to provide an optimistic estimate, and to then multiply by 2 to make it "realistic".

2. On top of that, add another x2 (which can be recalibrated as you learn how accurate this tech team is over time with estimates). Don't tell the developers about this, but make sure your higher-ups understand that this is what they need to be prepared for in terms of budgeting and time limits.

The reason you don't ask the developers to multiply by 4 directly, is to keep them motivated to aim for the x2, and avoid slacking or over engineering while feeling overly comfortable early on.

But by having the extra x2 in reserve, your back is covered, and you can afford to be cheritable with the dev team as they (as usually happens) go a bit over the x2 estimate.

This buys some early goodwill that can later be traded back in if you need them to up their game later on.

The alternative to the above is to exclusively find managers (at all levels) that can combine manager skills with high level engineering skills. Such managers often have the ability to expose unneccery delays directly, which includes the ability to tell apart delays caused by devs slacking from incompetence, scope creep or unexpected but valid causes.

Such people are really hard to find, though, for most companies. But companies that manage to build such high level top to bottom tech lead cultures may certainly be able to go from the 4x back down to 2x or even 1x compared to companies with non-technical managers.

One basic truism in business is that "Everybody wants a discount".

When I was naive and believed that Agile was not a sinister micromanagement toolkit to mess with programmers, I tried to explain to people that about half of our estimates should overshoot and half undershoot or they are biased and that there should be more overshoots since there is no upper bound on how much time a task can take if the estimate is wrong.

Ye. No. The burndown chart shouls be as straight as possible.

as things stand what is my incentive to provide an accurate estimate? if no one can question my estimate and hold me to it (well perhaps they can question it but we as industry have successfully been able to convince everyone that these are just estimates, nothing else...) what is my incentive to be accurate? If like one of the commenters above can say "it'll be 2 to 3 weeks" there is an INSANE difference between 2 and 3 weeks, 33% difference. it's like coming to buy a house and agent says "this house is $200k or $300k but you sign here on the bunch of dotted lines and we'll tell you all about it eventually before you have to cut a check." It is good to be in this industry (and especially if you WFH) - say 2 to 3 weeks, finish in 2 and get a week of working on your wellness (or another job :) )

Not really with price, but when I've had my house built, the date was overshoot by about 30% too, because of various reasons, like having to stop for winter because some supplies were late by a week several times or my builders had to help teams at other places from time to time (because other teams were late too), not doing anything at my house sometimes for days. So even when building homes (something they do again and again) you can't really put exact estimates.

Ryan, if you are out there reading this - ty.

Reminds me of: Always Multiply Your Estimates by π (2021), https://news.ycombinator.com/item?id=28667174

> ... scope creep ...

  If you want to know what Tesla does right and most of us do wrong, it's this: they ship something small, as fast as they can. Then they listen. Then they make a decision. Then they stick to it. And repeat.

  They don't make decisions any better than we do. That's key. It's not the quality of the decisions that matters. Well, I mean, all else being equal, higher quality decisions are better. But even if your decisions aren't optimal, sticking to them, unless you're completely wrong, usually works better than changing them.

I guess 'estimation poker' is a way to counteract the obvious strategy to coast and look competent.

In poker you can also look good by underbidding your peers and then snatch the easy ones to look good while the scapegoats look bad.

The strategy need some social status or incubent code knowledge relative to the team though, to get the good tasks.

Where I mostly worked, managers cared about deadlines they could tell to external clients, which they really hated to miss. Early on, I didn't realize that, and gave my best guess. If I guessed the correct median, I was missing it 50% of the time, and managers kept getting mad at me.

So I switched to estimates I could meet 90% of the time, and on the slow 10% I worked extra hours to meet my estimate anyway. Managers were happy. If I told them it would be done by Tuesday, it would be done by Tuesday.

But it had enormous benefits beyond that. In almost 90% of cases, I had free time. Sometimes I'd admit to finishing early, but I also used that time to clean up technical debt, automate the tedious parts of my job, or advance my skills. After a while, I could give estimates as short as my old 50% estimates, and still beat them 90% of the time because I'd made my tasks so much easier. Less technical debt also meant the resulting code was less likely to have bugs.

After a while, it seemed to me that all the other devs were overworked and I had it easy. But management gave me raises, and when they got in a jam, I was the guy they called on to bail them out.

If you "look good" too often, your estimates will be distrusted. So, just play some videogames and look like you're a genius at estimating.

This is good advice, as devs tend to underestimate.

"We'd like to replace an excel table for some calculations with a dialog. Here's the template, how long do you need?" which sounds simple enough turns into:

1- Decypher what the example excel template developed by someone over 10 years even does.

2- Oh, there are actually 10 templates and manual actions that give the end result.

2.5- Oh, btw, we asked an external company about doing this for us a while back and they wanted 1kk euros, crazy right?

3- Oh, we also need to generate, send and track offers via the app with the ability to add comments and upload files related to the offer. We also want the user to be queried about what data he has on hand so that calculations he cannot complete are not offered/he's notified as to what else he needs to proceed.

4- Oh, we also need change tracking/audit logs for everything.

5- Oh, we also need to get data from this place, find a free API and also a way to get data out of this software here.

In comparison to that at my previous job the tasks were way smaller and clearer so I'd essentially give myself deadlines when talking to my manager by saying X and Y should be done by Z, A by B.

The only thing I can think of in this situation is to essentially make internal pseudo contracts regarding requirements but then I'm making a pseudo contract with someone 3 levels in the hierarchy above me who's also the person who can terminate me. It's not like that pseudo contract will be read by anyone besides us so it seems better to display lots of uncertainty. At least if you're senior you have more authority in discussion and don't really have to give a fuck since everyone's looking to hire senior devs + your downgrade is a normal dev position. From junior the downgrade seems to be testing or McDonalds and you get to redo junior.

To only assign blame to people/teams when they’ve guaranteed in writing that it would be so and so, avoiding the downsides.

And blaming the process when there were no such guarantees?

E.g. if blame is assigned when there's a written guarantee, why would anyone ever make a written guarantee?

And not trying to be obtuse, but I've only ever seen blameless cultures work in absolute. Compromises let back in all the nasty mal-incentives you see driving unproductive CYA behaviors.

It's not a lack of technical competence, it's a lack of introspection and managerial soft skills.

https://www.penguinrandomhouse.com/books/690632/weve-got-you...

Also doing bidding for those estimates in addition could mean that there might be strong incentives for a lot of corner cutting for certain tasks, etc. People will value short term gains over long term gains when there's such pressure.

To answer it personally, which of the two, 7 figure bonus or being fired, it'd be I'd quit. If someone is structuring the development of software based on this premise, then they are going to need a different kind of person than me. But I admit I'm probably an outlier here. I don't really work for the money, and my salary is enough, and I don't like undo pressure.

For arguments sake let's say the 7 figures is $1,000,000. To offer that kind of bonus the project is likely going to be a larger one. And I'm assuming my estimate is determining the deadline, so of course I'm making sure it's something I think I can achieve.

But then there are other significant problems with this structure and the likelihood of meeting the deadline, and, more importantly, generating good code and user experience.

- +5% (in ignoring the -5% as no one cares if you're early unless it creates some sort of QA burden) implies a narrow window. On a 6 month project that is ~6 days. Enough that personnel changes or other uncontrollable factors could lead to a missed deadline. One person getting fed up and leaving would be a huge problem.

- The specification would have to be really clear and agreed upon, since there is much at stake.

- Any changes, scope creep, customer requests, could change the development time, and you'd have to have some sort negotiating buffer built in since there is now so much at stake. Otherwise you're going to get literally everything rejected by the developer as they drive towards the deadline (maybe that's what you want, though).

- Is the result worth having? A focus on a deadline, in my experience, tends to shortchange quality. But maybe the deadline is more important than quality.

- And lastly, if that deadline is missed, or worse, something changes the scope of the project and the bonus is not awarded because that led to the deadline being missed, you're going to have some super pissed developers that will not trust such an arrangement in the future.

I suspect you're talking about situations beyond my pay-grade. I've been a meat and potatoes programmer working in e-commerce and integrations mostly, and we don't see 7 figure bonuses. We certainly have had can't miss deadlines that we mostly didn't miss, but mostly those deadlines were due to external factors (API deprecation mostly), or financial considerations, or lastly, arbitrary deadlines set by management. On the latter, those mostly got missed. But that was to be expected as they were not tied to reality.

And I'd second leetcrews comment below of, "...but it's not a sustainable approach for delivering features". Maybe this scenario works once or twice, but it seems like a terrible way to develop software.

But still, an interesting thought experiment.

Something like a new engineer disagreeing with a PM, a PM disagreeing with higher management, etc…

There’s many reasons why that would be a favourable choice.

And a major problem with group estimates is that given how much knowledge a person has about the code, the effective time will vary so much.

So dunno. I have no experiance as a team lead or manager.

I would probably not track task time at all as a manager. It would give the illusion of insight. Rather some loose percent worked by project per programmer.

Management in most places is rather interested in getting planning right, not making up numbers and then failing at achieving them. They often lack the training to get it right (both because we have non-engineers as managers, and because we give shit management training to the engineers that become managers), but "management quits listening" is just an excuse.

I'm in this thing for ~4 decades now, at a good number of companies of all sizes (3-200,000) and I've seen the "not listening and insisting on made up numbers" exactly once. I see however a lot of engineers refuse to even attempt to make reasonable estimates.

Second, always pad your estimates. If you have been in the industry longer than 6 months, you'll already know how "off" your estimates can be. Take the actual delivery date, divide that by the estimated date, and that's your multiplier.