LLM Inevitabilism

There may be an "LLM Winter" as people discover that LLMs can't be trusted to do anything. Look for frantic efforts by companies to offload responsibility for LLM mistakes onto consumers. We've got to have something that has solid "I don't know" and "I don't know how to do this" outputs. We're starting to see reports of LLM usage having negative value for programmers, even though they think it's helping. Too much effort goes into cleaning up LLM messes.

> Look for frantic efforts by companies to offload responsibility for LLM mistakes onto consumers.

Not just by companies. We see this from enthusiastic consumers as well, on this very forum. Or it might just be astroturfing, it's hard to tell.

The mantra is that in order to extract value from LLMs, the user must have a certain level of knowledge and skill of how to use them. "Prompt engineering", now reframed as "context engineering", has become this practice that separates anyone who feels these tools are wasting their time more than they're helping, and those who feel that it's making them many times more productive. The tools themselves are never the issue. Clearly it's the user who lacks skill.

This narrative permeates blog posts and discussion forums. It was recently reinforced by a misinterpretation of a METR study.

To be clear: using any tool to its full potential does require a certain skill level. What I'm objecting to is the blanket statement that people who don't find LLMs to be a net benefit to their workflow lack the skills to do so. This is insulting to smart and capable engineers with many years of experience working with software. LLMs are not this alien technology that require a degree to use correctly. Understanding how they work, feeding them the right context, and being familiar with the related tools and concepts, does not require an engineering specialization. Anyone claiming it does is trying to sell you something; either LLMs themselves, or the idea that they're more capable than those criticizing this technology.

A couple of typical comments about LLMs would be:

"This LLM is able to capably output useful snippets of code for Python. That's useful."

and

"I tried to get an LLM to perform a niche task with a niche language, it performed terribly."

I think the right synthesis is that there are some tasks the LLMs are useful at, some which they're not useful at; practically, it's useful to be able to know what they're useful for.

Or, if we trust that LLMs are useful for all tasks, then it's practically useful to know what they're not good at.

Even if that's true, they are still not reliable. The same question can produce different answers each time.

This isn't really true when you control the stack, no? If you have all of your parameters set to be reproducible (e.g. temp 0, same seed), the output should be the same as long as everything further down the stack is the same, no?