←back to thread

Test-driven development with an LLM for fun and profit

(blog.yfzhou.fyi)
219 points crazylogger | 4 comments | 16 Jan 25 15:30 UTC | HN request time: 0s | source
Show context
xianshou ◴[16 Jan 25 17:59 UTC] No.42728570[source]▶
One trend I've noticed, framed as a logical deduction:

1. Coding assistants based on o1 and Sonnet are pretty great at coding with <50k context, but degrade rapidly beyond that.

2. Coding agents do massively better when they have a test-driven reward signal.

3. If a problem can be framed in a way that a coding agent can solve, that speeds up development at least 10x from the base case of human + assistant.

4. From (1)-(3), if you can get all the necessary context into 50k tokens and measure progress via tests, you can speed up development by 10x.

5. Therefore all new development should be microservices written from scratch and interacting via cleanly defined APIs.

Sure enough, I see HN projects evolving in that direction.

replies(12): >>42729039 #>>42729413 #>>42729713 #>>42729788 #>>42730016 #>>42730842 #>>42731468 #>>42733881 #>>42735489 #>>42736464 #>>42740025 #>>42747244 #
Arcuru ◴[16 Jan 25 18:33 UTC] No.42729039[source]▶
> 5. Therefore all new development should be microservices written from scratch and interacting via cleanly defined APIs.

Not necessarily. You can get the same benefits you described in (1)-(3) by using clearly defined modules in your codebase, they don't need to be separate microservices.

replies(4): >>42729689 #>>42737545 #>>42738887 #>>42744790 #
1. sdesol ◴[16 Jan 25 19:23 UTC] No.42729689[source]▶
Agreed. If the microservice does not provide any value from being isolated, it is just a function call with extra steps.
replies(1): >>42731310 #
2. __MatrixMan__ ◴[16 Jan 25 21:44 UTC] No.42731310[source]▶
I think the argument is that the extra value provided is a small enough context window for working with an LLM. Although I'd suggest making it a library if one can manage, that gives you the desired context reduction bounded by interfaces without taking on the complexities of adding an additional microservice.

I imagine throwing a test at an LLM and saying:

> hold the component under test constant (as well as the test itself), and walk the versions of the library until you can tell me where they're compatible and where they break.

If you tried to do that with a git bisect and everything in the same codebase, you'd end up varying all three (test, component, library) which is worse science than holding two constant and varying the third would be.

replies(1): >>42732302 #
3. sdesol ◴[16 Jan 25 23:32 UTC] No.42732302[source]▶
> I think the argument is that the extra value provided is a small enough context window for working with an LLM.

I'm not sure moving something that could work as function to a microservice would save much context. If anything, I think you are adding more context, since you would need to talk about the endpoint and having it route to the function that does what you need. When it is all over, you need to describe what the input and output is.

replies(1): >>42738444 #
4. __MatrixMan__ ◴[17 Jan 25 15:21 UTC] No.42738444{3}[source]▶
Oh certainly. I was arguing that if you need more isolation than a function gives you, don't jump to the conclusion that you need a service. Consider a library as a middle ground.