Naur's "Programming as Theory Building" and LLMs replacing human programmers

The Chinese Room argument is a great thought experiment for understanding why the computational model is an inadequate explanation of consciousness and qualia. But it proves nothing about reason, which LLMs have clearly shown needs to be distinguished from consciousness. And theories fall into the category of reason, not of consciousness. Or another way of putting it that you might find more acceptable: maybe a computer will never, internally, know that it has developed a theory - but it sure seems like it will be able to act and talk as if it had, much like a philosophical zombie.

> it proves nothing about reason, which LLMs have clearly shown needs to be distinguished from consciousness.

Uh, they have? Are you saying they know how to reason? Because if so, why is it that when I give a state of the art model documentation lacking examples for a new library and ask it to write something, it cannot even begin to do that, even if the documentation is in the training data? A model that can reason should be able to understand the documentation and create novel examples. It cannot.

This happened to me just the other day. If the model can reason, examples of the language, which it has, and the expository documentation should have been sufficient.

Instead, the model repeatedly inserted bullshitted code in the style of the language I wanted, but with library calls and names based on a version of the library for another language.

This is evidence of reasoning ability? Claude Sonnet 3.7 and Gemini Pro both exhibited this behavior last week.

I think this technology is fundamentally the same as it has been since GPT2

Absolutely LLMs can reason. There are limitations on their ability to reason, as you and everyone else has discovered. But they can absolutely reason about both concepts and the physical world in ways that, say, animals can't - even though presumably animals have at least some sort of self-consciousness and LLM's do not.

> A model that can reason should be able to understand the documentation and create novel examples. It cannot.

That's due to limitations imposed for "security". "Here's a new X, do Y with it" can result in holes bigger and more complex than anyone can currently handle "in time".

It's not about "abilities" with LLMs for now, but about functions that work within the range of edge cases, sometimes including them, some other times not.

You could still guide it to fulfill the task, though. It just cannot be allowed to do it on it's own but since just "forbidding" an LLM to do something is about as effective as doing that to a child with mischievous older brothers, the only ways to actually do it result in "bullshitted" code and "hallucinations".

If I understood the problem correctly, that is.

> The Chinese Room argument is a great thought experiment for understanding why the computational model is an inadequate explanation of consciousness and qualia.

To be as accurate as possible with respect to the primary source [0], the Chinese room thought experiment was devised as a refutation of "strong AI," or the position that

    the appropriately programmed computer really is a mind, in the
    sense that computers given the right programs can be literally
    said to understand and have other cognitive states.

Searle's position?

    Rather, whatever purely formal principles you put into the
    computer, they will not be sufficient for understanding, since
    a human will be able to follow the formal principles without
    understanding anything. [...] I will argue that in the literal
    sense the programmed computer understands what the car and the
    adding machine understand, namely, exactly nothing.

[0] https://home.csulb.edu/~cwallis/382/readings/482/searle.mind...

> maybe a computer will never, internally, know that it has developed a theory

Happens to people all the time :) ... especially if they don't have a concept of theories and hypotheses.

People are dumb and uneducated only until they aren't anymore, which is, even in the worst cases, no more than a decade of effort put in time. In fact, we don't even know how crazy fast neuro-genesis and or cognitive abilities might increase when a previously dense person reaches or "breaks through" a certain plateau. I'm sure there is research, but this is not something a satisfyingly precise enough answer can be formulated for.

If I formulate a new hypothesis, the LLM can tell me, "nope, you are the only idiot believing this path is worth pursuing". And if I go ahead, the LLM can tell me: "that's not how this usually works, you know", "professionals do it this way", "this is not a proof", "this is not a logical link", "this is nonsense but I commend your creativity!", all the way until the actual aha-moment when everything fits together and we have an actual working theory ... in theory.

We can then analyze the "knowledge graph" in 4D and the LLM could learn a theory of what it's like to have a potential theory even though there is absolutely nothing that supports the hypothesis or it's constituent links at the moment of "conception".

Stay put, it will happen.

> The Chinese Room argument is a great thought experiment for understanding why the computational model is an inadequate explanation of consciousness and qualia. But it proves nothing about reason

I think you misunderstand the Chinese Room argument [0]. It is exactly about how a mechanical process can produce results without having to reason.

[0] https://plato.stanford.edu/entries/chinese-room/

I imagine Searle feels vindicated since LLMs are good at translating Chinese.

On the other hand I am reminded of Nilsson's rebuttal:

> For all I know, Searle may only be behaving as if he were thinking deeply about these matters. But, even though I disagree with him, his simulation is pretty good, so I’m willing to credit him with real thought.

Nilsson's complaint that Searle is conflating the running program with the underlying system/interpreter that runs it seems accurate.