Naur's "Programming as Theory Building" and LLMs replacing human programmers

> First, you cannot obtain the "theory" of a large program without actually working with that program...

> Second, you cannot effectively work on a large program without a working "theory" of that program...

I find the whole argument and particularly the above to be a senseless rejection of bootstrapping. Obviously there was a point in time (for any program, individual programmer and humanity as a whole) that we didn't have a "theory" and didn't do the work, but now we have both, so a program and its theory can appear "de novo".

So with that in mind, how can we reject the possibility that as an AI Agent (e.g. Aider) works on a program over time, it bootstraps a theory?

> So with that in mind, how can we reject the possibility that as an AI Agent (e.g. Aider) works on a program over time, it bootstraps a theory?

Lack of effective memory, that might have worked if you constantly retrained the LLM incorporating the new wisdom iteratively like a human does, but current LLM architecture doesn't enable that. The context provided is neither large enough nor can it use it effectively enough for complex problems.

And this isn't easy to solve, you very quickly collapse the LLM if you try to do this in the naive ways. We need some special insight that lets us update LLM continuously as it works in a positive direction the way humans can.

Yeah, that's a good point. I absolutely agree that it needs access to effective long-term memory, but it's unclear to me that we need some "special insight". Research is relatively early on this, but we already see significant sparks of theory-building using basic memory retention, when Claude and Gemini are asked to play Pokemon [0][1]. It's clearly not at the level of a human player yet, but it (particularly Gemini) is doing significantly better than I expected at this stage.

[0] https://www.twitch.tv/claudeplayspokemon

[1] https://www.twitch.tv/gemini_plays_pokemon