←back to thread

Has the cost of building software dropped 90%?

(martinalderson.com)
419 points martinald | 4 comments | 08 Dec 25 19:00 UTC | HN request time: 0.805s | source
Show context
Volundr ◴[09 Dec 25 14:28 UTC] No.46205257[source]▶
FTA

> I've had Claude Code write an entire unit/integration test suite in a few hours (300+ tests) for a fairly complex internal tool. This would take me, or many developers I know and respect, days to write by hand.

I have no problem believing that Claude generated 300 passing tests. I have a very hard time believing those tests were all well thought out, consise, actually testing the desired behavior while communicating to the next person or agent how the system under test is supposed to work. I'd give very good odds at least some of those tests are subtly testing themselves (ex mocking a function, calling said function, then asserting the mock was called). Many of them are probably also testing implementation details that were never intended to be part of the contract.

I'm not anti-AI, I use it regularly, but all of these articles about how crazy productive it is skip over the crazy amount of supervision it needs. Yes, it can spit out code fast, but unless your prepared to spend a significant chunk of that 'saved" time CAREFULLY (more carefully than with a human) reviewing code, you've accepted a big drop in quality.

replies(7): >>46205349 #>>46205526 #>>46205624 #>>46206683 #>>46206705 #>>46208955 #>>46214506 #
jf22 ◴[09 Dec 25 15:00 UTC] No.46205624[source]▶
> you've accepted a big drop in quality.

Right, but you do it in a 10th of the time.

replies(2): >>46205955 #>>46206771 #
bagacrap ◴[09 Dec 25 16:21 UTC] No.46206771[source]▶
I mean, just say you view unit testing as nothing more than a checkbox.
replies(1): >>46207181 #
1. jf22 ◴[09 Dec 25 16:48 UTC] No.46207181[source]▶
I don't know why you'd think that.

200 decent unit tests are better than zero unit tests.

replies(2): >>46209757 #>>46245343 #
2. welshwelsh ◴[09 Dec 25 19:54 UTC] No.46209757[source]▶
The main benefit of writing tests is that is forces the developer to think about what they just wrote and what it is supposed to do. I often will find bugs while writing tests.

I've worked on projects with 2,000+ unit tests that are essentially useless, often fail when nothing is wrong, and rarely detect actual bugs. It is absolutely worse than having 0 tests. This is common when developers write tests to satisfy code coverage metrics, instead of in an effort to make sure their code works properly.

replies(1): >>46210712 #
3. jf22 ◴[09 Dec 25 21:11 UTC] No.46210712[source]▶
Look, you tell the LLMs what kind of tests you want and judge the quality before committing.

If you're letting the LLM create useless test that's on you.

I think you're reading these comments in bad faith as if I'm letting the LLM add slop to satisfy a metric.

No, I'm using an LLM to write good tests that I will personally approve as usefull, and other people will review too, before merging into master.

4. bagacrap ◴[12 Dec 25 15:58 UTC] No.46245343[source]▶
I don't think zero unit tests is the right answer either. And if you actually take the time to read all 300 and cull the useless or overlapping ones, you've invested much more than 10% of the time it would have taken you.

Having a zillion unit tests (of questionable quality) is a huge pita when you try to refactor.

When I am writing unit tests (or other tests), I'm thinking about all the time I'll save by catching bugs early -- either as I write the test or in the future as regressions crop up. So to place too much importance on the amount of time invested now is missing the point, and makes me think that person is just going through the motions. Of course if I'm writing throwaway code or a POC, I'll probably skip writing tests at all.

In order to add coverage for scenarios that I haven't even thought of, I prefer fuzz testing. Then I get a lot more than 2-300 tests and I don't even pretend to spend time reviewing the tests until they fail.

If you want to use an LLM to help expedite the typing of tests you have thought of, fine. If you just tell it to write the suite for itself, that's equivalent to hiring a (mediocre to bad) new grad and forcing them to write tests for you. If that's as good of an outcome as doing it yourself, I can only assume you are brand new to software engineering.