Some thoughts on LLMs and software development

In my company I feel that we getting totally overrun with code that's 90% good, 10% broken and almost exactly what was needed.

We are producing more code, but quality is definitely taking a hit now that no-one is able to keep up.

So instead of slowly inching towards the result we are getting 90% there in no time, and then spending lots and lots of time on getting to know the code and fixing and fine-tuning everything.

Maybe we ARE faster than before, but it wouldn't surprise me if the two approaches are closer than what one might think.

What bothers me the most is that I much prefer to build stuff rather than fixing code I'm not intimately familiar with.

"but quality is definitely taking a hit now that no-one is able to keep up."

And its going to get worse! So please explain to me how in the net, you are going to be better off? You're not.

I think most people haven't taken a decent economics class and don't deeply understand the notion of trade offs and the fact there is no free lunch.

Yep, my strong feeling is that the net benefit of all of this will be zero. The time you have to spend holding the LLM hand is almost equal to how much time you would have spent writing it yourself. But then you've got yourself a codebase that you didn't write yourself, and we all know hunting bugs in someone else's code is way harder than code you had a part in designing/writing.

People are honestly just drunk on this thing at this point. The sunken cost fallacy has people pushing on (ie. spending more time) when LLMs aren't getting it right. People are happy to trade convenience for everything else, just look at junk food where people trade in flavour and their health. And ultimately we are in a time when nobody is building for the future, it's all get rich quick schemes: squeeze then get out before anyone asks why the river ran dry. LLMs are like the perfect drug for our current society.

Just look at how technology has helped us in the past decades. Instead of launching us towards some kind of Star Trek utopia, most people now just work more for less!

Only when purely vibe coding. AI currently saves a LOT of time if you get it to generate boilerplate, diagnose bugs, or assist with sandboxed issues.

The proof is in the pudding. The work I do takes me half as long as it used to and is just as high in quality, even though I manage and carefully curate the output.

I use AI for most of those things. And I think it probably saves me a bit of time.

But in that study that came out a few weeks ago where they actually looked at time saved, every single developer overestimated their time saved. To the point where even the ones who lost time thought they saved time.

LLMs are very good at making you feel like you’re saving time even when you aren’t. That doesn’t mean they can’t be a net productivity benefit.

But I’d be very very very surprised if you have real hard data to back up your feelings about your work taking you half as long and being equal quality.

I mean, I used to average 2 hours of intense work a day and now it’s 1 hour.

How are you tracking that? Are you keeping a log, or are you just guessing? Do you have a mostly objective definition of intense work or are you just basing it on how you feel? Is your situation at work otherwise exactly the same, or have you gotten into a better groove with your manager? Are you working on exactly the same thing? Have you leveled up with some more experience? Have you learned the domain better?

Is your work objectively the same quality? Is it possible that you are producing less but it’s still far above the minimum so no one has noticed? Is your work good enough for now, but a year from now when someone tries to change it, it will be a lot harder for them?

Based on the only real studies we have, humans grossly overestimate AI time savings. It’s highly likely you are too.

_sigh_. Really dude? Just because people overestimate them on average doesn’t mean every person does. In fact, you should be well versed enough about the statistics to understand that it will be a spectrum that is highly dependent on both a persons role and how they use it.

For any given new tool, a range of usefulness that depends on many factors will affect people differently as individuals. Just because a carpenter doesn’t save much time because Microsoft excel exists doesn’t mean it’s not a hugely useful tool, and doesn’t mean it doesn’t save a lot of time for accountants, for example.

Instead of trying to tear apart my particular case, why not entertain the possibility that it’s more likely I’m reporting pretty accurately but it’s just I may be higher up that spectrum - with a good combo of having a perfect use case for the tool and also using the tool skilfully?

> _sigh_. Really dude? Just because people overestimate them on average doesn’t mean every person does.

In the study, every single person overestimated time saved on nearly every single task they measured.

Some people saved time, some didn’t. Some saved more time, some less. But every single person overestimated time saved by a large margin.

I’m not saying you aren’t saving time, but it’s very unlikely that if you aren’t tracking things very carefully that you are overestimating.

I’ll admit it’s possible my estimates are off a bit. What isn’t up for debate though is that it’s made a huge difference in my life and saved me a ton of time.

The fact that people overestimate its usefulness is somewhat of a “shrug” for me. So long as it _is_ making big differences, that’s still great whether people overestimate it or not.

If people overestimate time saved by huge margins, we don’t know whether it’s making big differences or not. Or more specifically whether the boost is worth the cost (both monetary and otherwise).

Only if we’re only using people’s opinions as data. There are other ways to do this.

Sure and if we look at data, the. only independent studies we have show either small productivity gains or a reduction in productivity for everything but small greenfield projects.

Studies plural? Can you link them?

Google for the Stanford study by Yegor Denisov-Blanch. You might have to pay to access the paper, but you can watch the author’s synopsis on YouTube.

For low complexity greenfield projects (best case) they found a 30% to 40% productivity boost.

For high-complexity brownfield projects (worst case) they found a -5% to 10% productivity boost.

The METR study from a few weeks ago showed an average productivity drop around 20%.

That study also found that the average developer believed AI had made them 20% more productive. The difference in perception and reality was on average 40 percentage points.