Measuring the impact of AI on experienced open-source developer productivity

This study neglects to incorporate the fact that I have forgotten how to write code.

I'm curious what space people are working in where AI does their job entirely.

I can use it for parts of code, algorithms, error solving, and maybe sometimes a 'first draft'.

But there is no way I could finish an entire piece of software with AI only.

Honestly, this is a fair point -- and speaks the difficulty of figuring out the right baseline to measure against here!

If we studied folks with _no_ AI experience, then we might underestimate speedup, as these folks are learning tools (see a discussion of learning effects in section (C.2.7) - Below-average use of AI tools - in the paper). If we studied folks with _only_ AI experience, then we might overestimate speedup, as perhaps these folks can't really program without AI at all.

In some sense, these are just two separate and interesting questions - I'm excited for future work to really dig in on both!

Not a lot of people are empowered to create an entire piece of software. Most are probably in the trenches squashing tickets.

I do create entire pieces of software, and while my workflow is always evolving, it goes something like this:

Define schemas, interfaces, and perhaps some base classes that define the attributes I'm thinking about.

Research libraries that support my cause, and include them.

Reference patterns I have established in other parts of the codebase; internal tooling for database, HTTP services, etc.

Instruct the agent to come up with a plan for a first pass at execution in markdown format. Iterate on this plan; "what about X?"

Splat a bunch of code down that supports the structure I'm looking for. Iterate. Cleanup. Iterate. Implement unit tests and get them to pass.

Go back through everything manually and adjust it to suit my personal style, while at the same time fully understanding what's being done and why.

I use STT a lot to have conversations with the agent as we go, and very rarely allow it to make sequential edits without reviewing first; this is a great opportunity to go back and forth and refine what's being written.

You are going well above and beyond what a lot of people do to be fair. There are people in senior roles who are just futzing with json files.

I think the question still stands.