←back to thread

AI should only run as fast as we can catch up

(higashi.blog)
196 points yuedongze | 2 comments | 08 Dec 25 17:38 UTC | HN request time: 0.4s | source
Show context
gradus_ad ◴[08 Dec 25 17:51 UTC] No.46195373[source]▶
The proliferation of nondeterministically generated code is here to stay. Part of our response must be more dynamic, more comprehensive and more realistic workload simulation and testing frameworks.
replies(5): >>46195431 #>>46195733 #>>46197437 #>>46197956 #>>46199307 #
yuedongze ◴[08 Dec 25 17:56 UTC] No.46195431[source]▶
i've seen a lot of startups that use AI to QA human work. how about the idea of use humans to QA AI work? a lot of interesting things might follow
replies(6): >>46195474 #>>46195546 #>>46195718 #>>46195741 #>>46195828 #>>46199496 #
colechristensen ◴[08 Dec 25 18:27 UTC] No.46195828[source]▶
Yes, but not like what you think. Programmers are going to look more like product managers with extra technical context.

AI is also great at looking for its own quality problems.

Yesterday on an entirely LLM generated codebase

Prompt: > SEARCH FOR ANTIPATTERNS

Found 17 antipatterns across the codebase:

And then what followed was a detailed list, about a third of them I thought were pretty important, a third of them were arguably issues or not, and the rest were either not important or effectively "this project isn't fully functional"

As an engineer, I didn't have to find code errors or fix code errors, I had to pick which errors were important and then give instructions to have them fixed.

replies(2): >>46196151 #>>46197630 #
1. manmal ◴[08 Dec 25 21:04 UTC] No.46197630[source]▶
Yeah, don‘t rely on the LLM finding all the issues. Complex code like Swift concurrency tooling is just riddled with issues. I usually need to increase to 100% line coverage and then let it loop on hanging tests until everything _seems_ to work.

(It’s been said that Swift concurrency is too hard for humans as well though)

replies(1): >>46199237 #
2. colechristensen ◴[08 Dec 25 23:39 UTC] No.46199237[source]▶
I don't trust programmers to find all the issues either and in several shops I've been in "we should have tests" was a controversial argument.

A good software engineering system built around the top LLMs today is definitely competitive in quality to a mediocre software shop and 100x faster and 1000x cheaper.