Measuring the impact of AI on experienced open-source developer productivity

(metr.org)

688 points dheerajvs | 1 comments | 10 Jul 25 16:29 UTC | HN request time: 0s | source

Show context

nestorD ◴[10 Jul 25 17:42 UTC] No.44523519[source]▶

One thing I could not find on a cursory read is how used were those developers to AI tools. I would expect someone using those regularly to benefit while someone who only played with them a couple of time would likely be slowed down as they deal with the friction of learning to be productive with the tool.

replies(1): >>44523729 #

uludag ◴[10 Jul 25 17:59 UTC] No.44523729[source]▶

>>44523519 #

In this case though you still wouldn't necessarily know if the AI tools had a positive causal effect. For example, I practically live in Emacs. Take that away and no doubt I would be immensely less effective. That Emacs improves my productivity and without it I am much worse in no way implies that Emacs is better than the alternatives.

I feel like a proper study for this would involve following multiple developers over time, tracking how their contribution patterns and social standing changes. For example, take three cohorts of relatively new developers: instruct one to go all in on agentic development, one to freely use AI tools, and one prohibited from AI tools. Then teach these developers open source (like a course off of this book: https://pragprog.com/titles/a-vbopens/forge-your-future-with...) and have them work for a year to become part of a project of their choosing. Then in the end, track a number of metrics such as leadership position in community, coding/non-coding contributions, emotional connection to project, social connections made with community, knowledge of code base, etc.

Personally, my prior probability is that the no-ai group would likely still be ahead overall.

replies(1): >>44537518 #

1. iLemming ◴[11 Jul 25 22:41 UTC] No.44537518[source]▶

>>44523729 #

FWIW, LLM tooling for Emacs is great. gptel for example allows you to converse with wide-range of different models from anywhere in Emacs — you can spontaneously send requests while typing some text or even browsing M-x menu. I often do things like "summarize current paragraph in pdf document" or "create a few anki cards based on this web page content", etc.

↑