The thing is, this feature leaned on every bit of experience and wisdom we had as a team --things like making sure the model is right, making sure the system makes sense overall and all the pieces fit together properly.
I don't know that "4x" is how it works --in this case, the AI let us really tap into the experience and skill we already had. It made us faster, but if we were missing the experience and wisdom part, we'd just be more prolific at creating messes.
I find AI helpful but no where near a multiplier in my day to day development experience. Converting a csv to json or vis-versa great, but AI writing code for me has been less helpful. Beyond boiler plate, it introduces subtle bugs that are a pain in the ass to deal with. For complicated things, it struggles and does too much and because I didn't write it I don't know where the bad spots are. And AI code review often gets hung up on nits and misses real mistakes.
So what are you doing and what are the resources you'd recommend?
Makes it seem like the actual problem to be solved is reducing the amount of boilerplate code that needs to be written, not using an LLM to do it.
I'm not smart enough to write a language or modify one, so this opinion is strongly spoken, weakly held.
To give a concrete example: I'm pretty good at doing Python coding on a whiteboard, because that's what I practiced for job interviews, and when I first learned Python I used Vim without setting up any Python integration.
I'm pretty terrible at doing Rust on a whiteboard, because I only learned it when I had a decent IDE and later even AI support.
Nevertheless, I don't think I'm a better programmer in Python.
Usually, companies benefit more from slowing down and prioritizing, not ‘going faster’.
Basically, I treat LLMs like a fairly competent unpaid intern and extend about the same level of trust to the output they produce.
It will still be a win: the rewards for the new productivity have to go somewhere in the economy.
Just like other productivity improvements in the past, it will likely be shared amongst various stakeholders depending on a variety of factors. The workers will get the lion's share.
What you will find is that the agent is much more successful in this regard.
The LLM has certain intrinsic abilities that match us and like us it cannot actually code 10,000 lines of code and have everything working in one go. It does better when you develop incrementally and verify each increment. The smaller the increments the better it performs.
Unfortunately the chain of thought process doesn’t really do this. It can come up with steps, sometimes the steps are too big and it almost never properly verifies things are working after each increment. That’s why you have to put yourself in the loop here.
Like allowing the computer to run test and verify an application works as expected on each step and to even come up with what verification means is a bit of what’s missing here and I think although this part isn’t automated yet, it can easily be automated where humans become less and less involved and distance themselves into a more and more supervisory role.
(https://www.oecd.org/en/data/indicators/hours-worked.html Mexico 2226 hours/week, US 1804, Denmark 1394)
The absolute best outcome of LLMs, and frankly where it seems to be headed, is the death of bloated one-stop-shop-for-everyone software. Instead people will be able to use their computers more directly than ever, without having to use/figure out complicated unintuitive swiss army knife software to solve their problems.
LLMs today can already make people the exact tools they need with no extra feature bloat or useless expansive packages. An print shop who just resizes their photos and does some minor adjustments not available from free tier software is no longer a slave to paying adobe $40/mo to use <1% of Photoshop's capabilities. They now can have their own tailor made in-house program for free.
LLM's will not be slotted in to replace devs on adobes dev teams. They can't work on a photoshop size codebase. However they will likely cut demand for Photoshop. Very few people will mourn the death of having to pay monthly for software just because there is a language barrier between them and their computer.
The first thing I'll note is that Claude Code with Claude 4 has been vastly better than everything else for me. Before that it was more like a 5-10% increase in productivity.
My workflow with Claude Code is very plain. I give it a relatively short prompt and ask it to create a plan. I iterate on the plan several times. I ask it to give me a more detailed plan. I iterate on that several times, then have Claude write it down and /clear to reset context.
Then, I'll usually do one or more "prototype" runs where I implement a solution with relatively little attention to code quality, to iron out any remaining uncertainties. Then I throw away that code, start a new branch, and implement it again while babysitting closely to make sure the code is good.
The major difference here is that I'm able to test out 5-10 designs in the time I would normally try 1 or 2. So I end up exploring a lot more, and committing better solutions.
Sometimes I´ll even go a bit crazy on this planning thing and do things a bit similar to what this guy shows: https://www.youtube.com/watch?v=XY4sFxLmMvw I tend to steer the process more myself, but typing whatever vague ideas are in my mind and ending up in minutes with a milestone and ticket list is very enabling, even if it isn´t perfect.
I also do more "drive by" small improvements:
- Annoying things that weren't important enough for a side quest writing a shell script, now have a shell script or an ansible playbook.
- That ugly CSS in an internal tool untouched for 5 years? fixed in 1 minute.
- The small prototype put into production with 0 documentation years ago? I ask an agentic tool to provide a basic readme and then edit it a bit so it doesn´t lie, well worth 15 minutes.
I also give it a first shot at finding the cause of bugs/problems. Most of the time it doesn't work, but in the last week it found right away the cause of some long standing subtle problems we had in a couple places.
I have also had sometimes luck providing it with single functions or modules that work but need some improvement (make this more DRY, improve error handling, log this or that...) Here I´m very conservative with the results because as you said it can be dangerous.
So am I more productive? I guess so, I don't think 4x or even 2x, I don't think projects are getting done much faster overall, but stuff that wouldn't have been done otherwise is being done.
What usually falls flat is trying to go on a more "vibe-coding" route. I have tried to come up with a couple small internal tools and things like that, and after promising starts, the agents just can't deal with the complexity without needing so much help that I'd just go faster by myself.
I think different generations of programmers have different opinions on what is quality output. Which makes judging the quality of code very context dependent.
If I were to guess I probably get somewhere in the range 10% to 20% productivity boost from LLMs. I think those are pretty astonishing numbers. The last time I got this kind of boost was when we got web search engines and sites like stack exchange.
I would suspect that if people experience 100% or more productivity boost from LLMs, something is off. Either we have very different ideas about quality, or we are talking about people who were not very productive to begin with.
I also think that LLMs are probably more useful if you are already a senior developer. You will have a better idea of what to ask for. You will also be in a better position to guide de LLM towards good answers.
...which kind of hints at my biggest worry: I think the gen-z programmers are facing a tough future. They'll have a harder time finding jobs with good mentors. They're faced with unrealistic expectations in terms of productivity. And they have to deal with the unrealistic expectations from "muggles" who understand neither AI nor programming. They will lack the knowledge to get the most from LLMs while having to deal with the expectation that they perform at senior levels.
We already see this in the job market. There has been a slight contraction and there are still a significant portion of senior developers available. Of course employers will prefer more experienced developers. And if younger developers believe in the hype that they can just vibe-code their way to success, this is just going to get worse.
Weird
Unless it goes to me I'm not entirely sure why I should care
I'll keep doing things the old way thanks, unless I personally get some benefit from it
"You are more productive but compensated the same" just shows how many of you are suckers
Sounds like AI has landed you on burnout treadmill
Statistics like https://media.equality-trust.out.re/uploads/2024/07/incomedi... suggest that since 1970 the top 10% have profited more in their income than the bottom 50%.