Large language models, small labor market effects [pdf]

Seems premature, like measuring the economic impact of the internet in 1985.

LLMs are more tech demo than product right now, and it could take many years for their full impact to become apparent.

I wouldn't call "premature" when llm companies ceos have been proposing ai agents for replacing workers - and similar things that I find debatable - in about the 2nd half of the twenties. I mean, a cold shower might eventually happen for a lot of Ai based companies

> cold shower might eventually happen for a lot of Ai based companies

undoubtedly.

The economic impact of some actually useful tools (Cursor, Claude) are propping up hundreds of billions of dollars in funding for, idk, "AI for <pick an industry> "or "replace your <job title> with our AI tool"

A big difference here is the sheer scale of investment. In 1985, the internet was running on the dreams of a few. The sheer depth of investment in "AI" currently is hard to fathom, and being injected into everything regardless of what customers want.

The most recent example is the Anthropic CEO:

> I think we will be there in three to six months, where AI is writing 90% of the code. And then, in 12 months, we may be in a world where AI is writing essentially all of the code

https://www.businessinsider.com/anthropic-ceo-ai-90-percent-...

This seems either wildly optimistic or comes with a giant asterisk that AI will write it by token predicting, then a human will have to double check and refine it.

I'm honestly slightly appalled by what we might miss by not reading the docs and just letting Ai code. I'm attending a course where we have to analyze medical datasets using up to ~200gb of ram. Calculations can take some time. A simple skim through the library (or even asking the chatbot) can tell you that one of the longest call can be approximated and it takes about 1/3rd of the time it takes with another solver. And yet, none of my colleagues thought about either looking the docs or asking the chatbot. Because it was working. And of course the chatbot was using the solver that was "standard" but that you probably don't need to use for prototyping.

Again. We had some parts of one of 3 datasets split in ~40 files, and we had to manipulate and save them before doing anything else. A colleague asked chatgpt to write the code to do it and it was single threaded, and not feasible. I hopped up on htop and upon seeing it was using only one core, I suggested her to ask chatgpt to make the conversion run on different files in different threads, and we basically went from absolutely slow to quite fast. But that supposed that the person using the code knows what's going on, why, and what is not going on. And when it is possible to do something different. Using it without asking yourself more about the context is a terrible use imho, but it's absolutely the direction that I see we're headed towards and I'm not a fan of it

I anticipate a control issue, where agents can produce code faster than people can analyze and beside applications with small visible surfaces, nobody will be able to check what is going on

I saw people with trouble manipulating boolean tables of 3 variables in their head trying to generate complete web applications, it will work for linear duties (input -> processing -> storage) but I highly doubt they will be able to understand anything with 2nd order effects

I wouldn't really pay attention to what CEOs say, it's their job to sell things and drum up investment.

No one really knows exactly how AI will play out. There's a lot of motivated reasoning going on, both from hypesters and cynics.

That's because the tech industry has more money than god and nothing better to do with it.

Microsoft alone has half a trillion dollars in assets, and Apple/Google/Meta/Amazon are in similar financial positions. Spending a few tens of billions on datacenters is, as crazy as it sounds, nothing to them.

While ARPANET itself reportedly cost somewhere between 10 - 20 million USD, which is relatively cheap, the precursor research that allowed the internet to take off - which is directed more at general computing and advanced computer networks, the telecommunications investments - cost many billions of dollars, it was mostly public money is the biggest difference.

That said, private companies are pumping alot of money into this space, but technological progress is a peaks and valley situation. I imagine most of the money will ultimately move the needle very little following things of dubious hindsight value

This quote to me feels more along the "no computer will ever need more than 640kb of RAM" lines in terms of historical accuracy. Like whoops, nope.

> I wouldn't really pay attention to what CEOs say, it's their job to sell things and drum up investment

We don't have a choice but to pay attention to what CEOs say, they are the ones that might lead our companies off of cliffs if we let them

That statement seems extremely hyperbolic. Just today I tried automating some very pedestrian code for a system that wasn't particularly well-documented and ChatGPT 4o hallucinated the entire API. It was deeply frustrating and wasted more of my time than it would have taken to just slog through the documentation.

I won't deny that LLMs can be useful--I still use them--but in my experience an LLM's success rate in writing working code is somewhere around 50%. That leads to a productivity boost that, while not negative, isn't anywhere near the wild numbers that are bandied about.

It's not my company, it can go off the cliff for all I care.

> people with trouble manipulating boolean tables of 3 variables in their head

To be fair, 3 booleans (2^3=8) is more working memory than most people are productive with. Way more if they’re nullable :)

Really feels like a race to the bottom to me. You used to have a trade that made you try to master those 8 configurations and now you're acting like the ignorant manager asking "agents" to deal with it.