This is the way LLM-enhanced coding should (and I believe will) go.
Treating the LLM like a compiler is a much more scalable, extensible and composable mental model than treating it like a junior dev.
replies(2):
A version that DID work like a compiler would be super interesting - it could replace the function body with generated Python code on your first call and then reuse that in the future, maybe even caching state on disk rather than in-memory.
I can see this eventually going in the direction of "bidirectional synchronization" of NL representation and code representation (similar to how jupytext allows you work with notebooks in browser or markdown in editor). But a single representation that's completely NL and deliberately throwing away a code representation sounds like it would be the opposite of productivity..
I would like to try something like this in Rust: - you use a macro to stub out the body of functions (so you just write the signature) - the build step fills in the code and caches it - on failures the, the build step is allowed to change the function bodies generated by LLMs until it satisfies the test / compile steps - you can then convert the satisfying LLM-generated function bodies into a hard code (or leave it within the domain of "changeable by the llm")
It sandboxes what the LLM can actually alter, and makes the generation happen in an environment where you can check right away if it was done correctly. Being Rust, you get a lot of more verifications. And, crucially, keeps you in the driver's seat.
Yeah, I do think that LLMs acting as compilers for super high-level specs (the new "code") is a much better approach than chatting with a bot to try to get the right code written. LLM-derived code should not be "peer" to human-written code IMO; it should exist at some subordinate level.
The fact that they're non-deterministic makes it a bit different from a traditional compiler but as you say, caching a "known good" artifact could work.
You can even pin the last result:
pinned function main() {
// Print "Hello World" to the console
}