Naur's "Programming as Theory Building" and LLMs replacing human programmers

Although I'm sympathetic to the author's argument, I don't think they've found the best way to frame it. I have two main objections i.e. points I guess LLM advocates might dispute.

Firstly:

> LLMs are capable of appearing to have a theory about a program ... but it’s, charitably, illusion.

To make this point stick, you would also have to show why it's not an illusion when humans "appear" to have a theory.

Secondly:

> Theories are developed by doing the work and LLMs do not do the work

Isn't this a little... anthropocentric? That's the way humans develop theories. In principle, could a theory not be developed by transmitting information into someone's brain patterns as if they had done the work?

> To make this point stick, you would also have to show why it's not an illusion when humans "appear" to have a theory.

This idea has already been explored by thought experiments such as John Searle's so-called "Chinese room" [0]; an LLM cannot have a theory about a program, any more than the computer in Searle's "Chinese room" understands "Chinese" by using lookup tables to generate canned responses to an input prompt.

One says the computer lacks "intentionality" regarding the topics that the LLM ostensibly appears to be discussing. Their words aren't "about" anything, they don't represent concepts or ideas or physical phenomena the same way the words and thoughts of a human do. The computer doesn't actually "understand Chinese" the way a human can.

[0] https://en.wikipedia.org/wiki/Chinese_room

You can state the argument formally as A has property B. Property B' implies property C. Hence A has property C. The fallacy is the sleight of hand where two almost but not quite identical properties B and B' are used, in this case two different defitions of theory, only one of which requires some ineffable mind consciousness.

It's important not to get caught up in a discussion about whether B or B' is the proper definition, but instead see that it's the inconsistency that is the issue.

LLM's build an internal representation that let's them efficiently and mostly successfully manipulate source code. Whether that internal representation is satisfies your criteria for a theory doesn't change that fact. What does matter to the highest degree however is where they succeed and where they fail, and how the representations and computing can improve the success rate and capabilities.

No, I don't agree with this formalization. It's more that (some) humans have a "theory" of the program (in the same sense used by Ryle and Naur); let's take for granted that if one has a theory, then they have understanding; thus (some) humans have an understanding of the program. It's not equivocating between B and B', but rather observing that B implies B'.

Thus, if an LLM lacks understanding (Searle), then they don't have a theory either.

> LLM's build an internal representation that let's them efficiently and mostly successfully manipulate source code. Whether that internal representation is satisfies your criteria for a theory doesn't change that fact.

The entire point of Naur's paper is that the activity of programming, of software engineering, is not just "manipulating source code." It is, rather, building a theory of the software system (which implies an understanding of it), in a way that an LLM or an AI cannot, as posited by Searle.