Counterpoint: the AI fanboys and AI companies, with all their insane funding couldn't come up with a better study and bigger sample size, because LLMs simply don't help experienced developers.
What follows is that the billion dollar companies just couldn't create a better study, either because they tried and didn't like the productivity numbers not being in their favor (very likely), or because they are that sloppy and vibey that they don't know how to make a proper study (I wouldn't be surprised, see ChatGPT's latest features: "study mode" which had a blog post! and you know that the level is not very high).
Again, until there is a better study, the consensus is LLMs are 19% productivity drain for experienced developers, and if they help certain developers, then most likely those developers are not experienced.
How's that for an interpretation?
There seems to be an attitude, or at least a pretended attitude, amongst the true believers that the heretics are dooming themselves, left behind in a glorious AI future. But the AI coding tools du jour are completely different from the ones a year ago! And in six months they'll be different again!
- Cause currently what we see in OSS is LLM trash. https://www.reddit.com/r/webdev/comments/1kh72zf/open_source...
- And a large majority of users don't want that copilot trash in their default github experience: https://www.techradar.com/pro/angry-github-users-want-to-dit...
At what point that trash will become gold? 5 more years? And if it doesn't, at what point trash stays trash?
- When there is a study showing that trash is actually sapping 19% of your performance? https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...
- When multiple studies show using it makes you dumb? https://tech.co/news/another-study-ai-making-us-dumb
Cause I am pretty sure NFT still has people who swear by them and say "just give it time". At what point can we confidently declare that NFTs are useless without the cultist fanbase going hurr durr? What about LLMs?