AI agents: Less capability, more reliability, please

I feel like OP would have been better of not referencing the viral thread about a developer not using any version control and surprised when the AI made changes, I don't think anyone who doesn't understand version control should be using a tool like cursor, there are other SAAS apps that build and deploy apps using AI and for people with the skill demonstrated in the thread, that is what they should be using.

It's like saying rm -rf / should have more safeguards built in. It feels unfair to call out the AI based tools for this.

`rm -rf /` does have a safeguard:

> For example, if a user with appropriate privileges mistakenly runs ‘rm -rf / tmp/junk’, that may remove all files on the entire system. Since there are so few legitimate uses for such a command, GNU rm normally declines to operate on any directory that resolves to /. If you really want to try to remove all the files on your system, you can use the --no-preserve-root option, but the default behavior, specified by the --preserve-root option, is safer for most purposes.

https://www.gnu.org/software/coreutils/manual/html_node/Trea...

I think it's a useful anecdote because it underscores how catastrophically unreliable* agents can be, especially in the hands of users who aren't experienced in the particular domain. In the domain of programming, it's much easier to quantify a "catastrophic" scenario vs. more open-ended "real world" situations like booking a flight.

* "unreliable" may not be the right word. For all we know, the agent performed admirably given whatever the user's prompt may have been. Just goes to show that even in a relatively constricted domain of programming, where a lot (but far from all) outcomes are binary, the room for misinterpretation and error is still quite vast.

That was added in 2006, so didn’t exist for a good half of its life (even longer if you count pre-GNU). I remember rm -rf / being considered just one instance of having to double-check what you do when using the -rf option. It’s one reason it became common to alias rm to rm -i.

Technically, they could be using version control, not have a copy on their local machine for some reason, and have an AI agent issue a `git push -f` wiping out all the previous work.

More than that, I think it's quite relevant, because it shows that the complexity in the tooling around writing code manually is not so inessential as it seems.

Any system capable of automating a complex task will by need be more complex than the task at hand. This complexity doesn't evaporate when you through statistical fuzzers at it.

I know this trope appears fairly often, but in reality unless it's someone's copy of $(git serve) running under their desk with no CI attached to it whatsoever, the commit history still exists and one can recover from a force push by just typing the equivalent of github.example.com/MyAwesomeOrg/MyRepo/commit/cafebabedeadbeef

I hypothesize that a $(git fetch --mirror) would pull down the "orphaned" revision, too, but don't currently have the mental energy to prove it

I actually had cursor on in yolo mode with the github integration on and it's actually pretty good about doing commits and pushes and opening PRs and stuff. Though I would never do that in a repo I cared a lot about. It doesn't always know what branch it's on and makes assumptions all the time about what org it's in, etc...