Two things that stand out to me are 1. it depends a lot on what kind of task you are having the LLM do. and 2. if the LLM process takes more time, it is very likely your cognitive effort was still way less - for sysadmin kinds of tasks, working with less often accessed systems, LLMs can read --help, man pages, doc sites, all for you, and give you the working command right there (And then run it, and then look at the output and tell you why it failed, or how it worked, and what it did). There is absolutely no question that second part is a big deal. Sticking it onto my large open source project to fix a deep, esoteric issue or write some subtle documentation where it doesnt really "get" what I'm doing, yeah it is not as productive in that realm and you might want to skip it for the thinking part there. I think everyone is trying to figure out this question of "when and how" for LLMs. I think the sweet spot is for tasks involving systems and technologies where you'd otherwise be spending a lot of time googling, stackoverflowing, reading man pages to get just the right parameters into commands and so forth. This is cognitive grunt work and the LLMs can do that part very well.
My week of effort with it was not really "coding on my open source project"; two examples were, 1. running a bunch of ansible playbooks that I wrote years ago on a new host, where OS upgrades had lots of snags; I worked with Claude to debug all the various error messages and places where the newer OS distribution had different packages, missing packages, etc. it was ENORMOUSLY helpful since I never look at these playbooks and I dont even remember what I did, Claude can read it for you and interpret it as well as you can. 2. I got a bugzilla for a fedora package that I packaged years ago, where they have some change to the directives used in specfiles that everyone has to make. I look at fedora packaging workflows once every three years. I told Claude to read the BZ and just do it. IT DID IT. I had to get involved running the "mock" suite as it needed sudo but Claude gave me the commands. zero googling. zero even reading the new format of the specfile (the bz linked to a tool that does the conversion). From bug received to bug closed and I didnt do any typing at all outside of the prompt. Had it done before breakfast since I didnt even need any glucose for mental energy expended. This would have been a painful and frustrating mental effort otherwise.
so the studies have to get more nuanced and survey a lot more than 16 devs I think