Naur's "Programming as Theory Building" and LLMs replacing human programmers

Wait, isn't the conclusion to take from the "Chinese room" literally the opposite of what you suggest? I.e. it's the most basic, go-to example of a larger system showing capability (here, understanding Chinese) that is not present in any of its constituent parts individually.

> Their words aren't "about" anything, they don't represent concepts or ideas or physical phenomena the same way the words and thoughts of a human do. The computer doesn't actually "understand Chinese" the way a human can.

That's very much unclear at this point. We don't fully understand how we relate words to concepts and meaning ourselves, but to the extent we do, LLMs are by far the closest implementation of those same ideas in a computer.

The Chinese room experiment was originally intended by Searle to (IIUC) do as you claim and justify computers as being capable of understanding like humans do. Since then, it has been used both in this pro-computer, "black box" sense and in the anti-computer, "white box" sense. Personally, I think both are relevant, and the issue with LLMs currently is not a theoretical failing but rather that they aren't convincing when viewed as black boxes (e.g. the Turing test fails).

> the conclusion to take from the "Chinese room"

We can hem and haw about whether or not there are others, but the particular conclusion I am drawing from is that computers lack "intentionality" regarding language, and indeed about anything at all. Symbol shunting, pencil pushing, and the mechanics of syntax are insufficient for the production of meaning and understanding.

That is, to oversimplify, the broad distinction drawn in Naur's article regarding the "programming as text manipulation" view vis-a-vis "programming as theory building."

> That's very much unclear at this point.

It's certainly a central point of contention.

The Chinese Room is a mirror that reflects people’s hidden (well, often not very, but still) biases about whether the universe is mechanical or whether understanding involves dualistic metaphysical woo back at them as conclusions.

That's not why it was presented, of course, Searle aimed at proving something, but his use of it just illustrates which side of that divide he was on.

No, it was used to argue that computers could pass the Turing test and still _not_ understand anything. It was a reducto intended to dispute exactly the claim you are ascribing to it, and argue _against_ "computers as being capable of understanding like humans do".

Thanks. I stand corrected. I guess I should also add that, aside from the black box view, there are pro-computer stances that claim there is mentality and intentionality.

Do you think gravity force is mechanical or is it a metaphysical woo? Because scientists have no idea how it works precisely, just like our brain and consciousness.

Hint - there are not only these two possibilities you have mentioned.

>We don't fully understand how we relate words to concepts and meaning ourselves,

This is definitely true.

>but to the extent we do, LLMs are by far the closest implementation of those same ideas in a computer

Well - this is half true but meaningless. I mean - we don't understand so LLM's are as good a bet as anything.

LLMs will confidently tell you that white wine is good with fish, but they have no experience of the taste of wine, or fish, or what it means for one to compliment the other. Humans all know what it's like to have fluid in their mouths, they know the taste of food and the feel of the ground under their feet. LLMs have no experience, they exist crystalised and unchanging in an abstract eternal now, so they literally can't understand anything.

I agree with your general point: it is a mistake to say "these two things are mysterious, therefore they are the same".

That said:

> LLMs have no experience, they exist crystalised and unchanging in an abstract eternal now, so they literally can't understand anything.

Being crystalised and unchanging, doesn't tell us either way if they do or don't "understand" anything — if it did, then I could only be said to "understand" whatever I am at some moment actually experiencing, so it would not be allowed to say, for example, that I can understand "water in my mouth" because my memory of previous times I had water in my mouth seem to be like that.

They're definitely not "like us", but that's about all I can say with confidence, and it's a very vague statement.

It's incoherent to think the ability to reason requires the reasoner to be able to change permanently. You realize that LLMs do change; their context window and model weights change on every processed token. Not to mention the weights can be saved and persisted in a sense via LORAs.

The belief LLMs cannot reason maybe justifiable for other reasons, just not for reasons you've outlined.

> Do you think gravity force is mechanical or is it a metaphysical woo? Because scientists have no idea how it works precisely, just like our brain and consciousness.

Nonsense. We know exactly how gravity works, with high precision. We don't know why it works.

I'm not sure you're right you know. I think that the way that an LLM maintains a conversation is to have the conversational thread fed into an instance of it at every step. You can see this if you do a conversation step by step and then take all of it (including the LLM responses) apart from the final outcome and paste that into a new thread:

https://chatgpt.com/share/6814e827-81cc-8001-a75f-64ed6df5fc...

https://chatgpt.com/share/6814e7fb-f4d0-8001-a503-9c991df832...

if you think about how these things work as services you can see that this makes sense. The model weights are several gb, so caching the model weights for utilisation by a particular customer is impractical. So if the forward pass does update the model then that's instantly discarded, what's retained is the conversational text, and that's the bit that's uploaded to the model on each iteration for a new reply. There are hundreds of requests pinging through the data center where the models are used every second, all of these use the same models.

But if you believe that there is a reasoning process taking place in the text then fair enough.