Measuring the impact of AI on experienced open-source developer productivity

> Interestingly, the best AI assisted devs have often moved to management/solution architecture, and they find the AI code tools brought back some of the love of coding

This suggests me though that they are bad at coding, otherwise they would have stayed longer. And I can't find anything in your comment that would corroborate the opposite. So what gives?

I am not saying what you say is untrue, but you didn't give any convincing arguments to us to believe otherwise.

Also, you didn't define the criteria of getting better. Getting better in terms of what exactly???

> This suggests me though that they are bad at coding, otherwise they would have stayed longer.

Or they care about producing value, not just the code, and realized they had more leverage and impact in other roles.

> And I can't find anything in your comment that would corroborate the opposite.

I didn’t try and corroborate the opposite.

Honestly, I don’t care about the “best coders.” I care about people who do their job well, sometimes that is writing amazing code but most of the time it isn’t. I don’t have any devs in my company who work in a magical vacuum where they are handed perfectly written tasks, they complete them, and then they do the next one.

If I did, I could replace them with AI faster.

> Also, you didn't define the criteria of getting better. Getting better in terms of what exactly?

Delivery velocity - bug fixes, features, etc. that pass testing/QA and goes to prod.

> Honestly, I don’t care about the “best coders.”

> Interestingly, the best AI assisted devs have often moved to management/solution architecture

Is it just me? Or does it seem to others as well that you pretty much rank these people even at the moment and your first comment contradicts your second comment? Especially when you admit that you rank them based on velocity.

I am not saying you shouldn't do that, but it feels to me like rating road construction workers on the number of potholes fixed, even though it's very possible that the potholes are caused by the sloppy work to begin with.

Not what I would want to do.

I'm not bad at coding. I would say I'm pretty damned good. But coding is a means-to-an-end. I come up with an idea, then I have the long-winded middle bit where I have to write all the code, spin up a DB, create the tables, etc.

LLMs have given me a whole new love of coding, getting rid of the dull grind and letting me write code an order of magnitude quicker than before.

> Is it just me? Or does it seem to others as well that you pretty much rank these people even at the moment and your first comment contradicts your second comment?

I think you are reading what you want to read and not what I said, so yes it is you. The most productive, valuable people with developer titles in my organizations are not the ones who write the cleanest, most beautiful, most perfect code. They do all of the other parts of the job well and write solid code.

Following the introduction of AI tools, many of the people in my organization who most effectively learned to use those tools are people who previously chose to move to manager and SA roles.

Not only are these not contradictory, they fit quite well together. People who do the things around coding well, but maybe have to work hard at writing the actual code, are better at using the AI tools than exceptional coders. For my organization, the former are generally more valuable than the latter without AI, and that is increasing as a result of AI.

> I am not saying you shouldn't do that, but it feels to me like rating road construction workers on the number of potholes fixed, even though it's very possible that the potholes are caused by the sloppy work to begin with.

Not if your measurement includes quality testing the pothole repairs, which mine does, as I explicitly called out. I work in industries with extensive, long testing cycles, we are (imperfectly, of course) able to measure productivity based on things which make it through those cycles.

You are trying very hard to find ways to ignore what I am saying. It is fine if you don’t want to believe me, but these things have been true based on our observations:

A. Great “coders” have a much harder time picking up AI dev tools and using them effectively, and when they see how others use them they will admit that isn’t how they use them. They will revert to their previous habits and give up on the tools.

B. The productivity gains for the people who are good at using the tools, as measured by velocity with a minimum bar for quality (with substantial QA), are very high.

C. We have measured these things to thoroughly understand the ROI and we are accelerating our investment in AI coding tools as a result.

Some caveats I am absolutely willing to make - we are not working on bleeding edge tech doing things no one has ever done before.

We failed to effectively use AI many times before we started to get it right.

There are developers who are slower with the AI code tools than without it.

I am not convinced.

If what you write was true, then the rate of bugs of those incredible devs would simply fall to zero at one point, and at that point they would become a legend who we all would have heard of by now. So the whole story sounds too fishy to my taste.

It's OK if you want to manage your team this way. Everyone needs some external feedback to confirm their own bias. It seems you found yours and it works for you.

It's just not a good argument in support of AI or AI assisted development.

It's too anecdotal.

And since you are the one who are telling me that you are right, and not others, it makes me even more skeptical about the whole story.