Nobody knows how to build with AI yet

That time dilation feels a bit like what METR reported:

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...

Developers believe they complete tasks 25% faster with AI but when measured they are 19% slower when using AI.

This was the study with a population size of 16?

It depends on how the AI is used; there's a huge difference in productivity between a structured workflow with repeated automated feedback to the AI, and just ad-hoc prompting. For instance, Gemini 2.5 coding up a spec-compliant HTTP2.0 server in two weeks: https://outervationai.substack.com/p/building-a-100-llm-writ... . 15k lines of code and 30k lines of tests; no human coder could produce something like that so fast.

Would that application not exist in the training data already?

It's cool that it made something that works, but that code is pretty bad. Trial and error is not the way to develop something like that.

The are around three Golang HTTP2.0 servers on Github, but it wasn't just regurgitating the application from memory, as if it was then it would have been mostly correct first try, and it wouldn't have needed to spend 80%+ of the development time in a code-compile-test cycle of fixing bugs identified by integration tests and spec conformance tests.

So you are saying that LLMs have no capacity to re-use existing work, and will just burn hours and money re-inventing wheels?

Welcome to The Future™!

That’s reminiscent of how when people are “in the flow”, they feel productive but also tend to not notice how quickly time passes.

If you tell it to reinvent the wheel, it will.

If you tell it to use existing libraries (and they are in its training data) it will do that instead.

If you tell it about libraries it hasn't encountered before it can use those instead.

16 developers doing hundreds of tasks. And the effect size was big enough for the result to be statistically significant.

Reminds me of that quaalude scene from the movie The Wolf of Wall Street.

Do you have any better research we can read?

I would not say that is ”how the AI is used”. That problem space is one where humans have spent unusually much time defining a spec and writing a test suite.

A million monkeys randomly typing could actually complete that task as well.

This was the only study that's been done so far?