←back to thread

Test-driven development with an LLM for fun and profit

(blog.yfzhou.fyi)
219 points crazylogger | 2 comments | 16 Jan 25 15:30 UTC | HN request time: 1.469s | source
Show context
xianshou ◴[16 Jan 25 17:59 UTC] No.42728570[source]▶
One trend I've noticed, framed as a logical deduction:

1. Coding assistants based on o1 and Sonnet are pretty great at coding with <50k context, but degrade rapidly beyond that.

2. Coding agents do massively better when they have a test-driven reward signal.

3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant.

4. From (1)-(3), if you can get all the necessary context into 50k tokens and measure progress via tests, you can speed up development by 10x.

5. Therefore all new development should be microservices written from scratch and interacting via cleanly defined APIs.

Sure enough, I see HN projects evolving in that direction.

replies(12): >>42729039 #>>42729413 #>>42729713 #>>42729788 #>>42730016 #>>42730842 #>>42731468 #>>42733881 #>>42735489 #>>42736464 #>>42740025 #>>42747244 #
Arcuru ◴[16 Jan 25 18:33 UTC] No.42729039[source]▶
> 5. Therefore all new development should be microservices written from scratch and interacting via cleanly defined APIs.

Not necessarily. You can get the same benefits you described in (1)-(3) by using clearly defined modules in your codebase, they don't need to be separate microservices.

replies(4): >>42729689 #>>42737545 #>>42738887 #>>42744790 #
1. lolinder ◴[17 Jan 25 13:57 UTC] No.42737545[source]▶
I wonder if we'll see a return of the kind of interface file present in C++, Ocaml, and Ada. These files, well commented, are naturally the context window to use for reference for a module.

Even if languages don't grow them back as a first class feature, some format that is auto generated from the code and doesn't include the function bodies is really what is needed here.

replies(1): >>42738366 #
2. senkora ◴[17 Jan 25 15:15 UTC] No.42738366[source]▶
Python (which I mention because it is the preferred language of LLM output) has grown stub files that would work for this:

https://peps.python.org/pep-0484/#stub-files

I guess that this usecase would be an argument to include docstrings in your Python stub files, which I hadn’t considered before.