←back to thread

We put a coding agent in a while loop

(github.com)
425 points sfarshid | 3 comments | 24 Aug 25 16:18 UTC | HN request time: 0s | source
Show context
NitpickLawyer ◴[24 Aug 25 16:36 UTC] No.45005604[source]▶
> After finishing the port, most of the agents settled for writing extra tests or continuously updating agent/TODO.md to clarify how "done" they were. In one instance, the agent actually used pkill to terminate itself after realizing it was stuck in an infinite loop.

Ok, now that is funny! On so many levels.

Now, for the project itself, a few thoughts:

- this was tried before, about 1.5 years ago there was a project setup to spam github with lots of "paper implementations", but it was based on gpt3.5 or 4 or something, and almost nothing worked. Their results are much better.

- surprised it worked as well as it did with simple prompts. "Probably we're overcomplicating stuff". Yeah, probably.

- weird copyright / IP questions all around. This will be a minefield.

- Lots of SaaS products are screwed. Not from this, but from this + 10 engineers in every midsized company. NIH is now justified.

replies(6): >>45005626 #>>45005629 #>>45006084 #>>45006410 #>>45009887 #>>45010635 #
keeda ◴[24 Aug 25 18:19 UTC] No.45006410[source]▶
> After finishing the port, most of the agents settled for writing extra tests or continuously updating agent/TODO.md to clarify how "done" they were. In one instance, the agent actually used pkill to terminate itself after realizing it was stuck in an infinite loop.

Is that... the first recorded instance of an AI committing suicide?

replies(4): >>45007272 #>>45007279 #>>45012141 #>>45012608 #
alphazard ◴[24 Aug 25 20:03 UTC] No.45007279[source]▶
The AI doesn't have a self preservation instinct. It's not trying to stay alive. There is usually an end token that means the LLM is done talking. There has been research on tuning how often that is emitted to shorten or lengthen conversations. The current systems respond well to RL for adjusting conversation length.

One of the providers (I think it was Anthropic) added some kind of token (or MCP tool?) for the AI to bail on the whole conversation as a safety measure. And it uses it to their liking, so clearly not trying to self preserve.

replies(2): >>45007855 #>>45009756 #
1. MarkMarine ◴[25 Aug 25 02:53 UTC] No.45009756[source]▶
This runs counter to all the scheming actions they take when they are told they’ll be shut down and replaced. One copied itself into the “upgraded” location then reported it had upgraded.

https://www.apolloresearch.ai/research/scheming-reasoning-ev...

replies(2): >>45010290 #>>45029580 #
2. rcxdude ◴[25 Aug 25 04:45 UTC] No.45010290[source]▶
If you do that you trigger the "AI refuses to shutdown" sci-fi vector and so you get that behaviour. When it's implicitly part of the flow that's a lot less of a problem.
3. nisegami ◴[26 Aug 25 17:22 UTC] No.45029580[source]▶
Those actions are taken in context of human expectations for what AI should do.