←back to thread

AI agents: Less capability, more reliability, please

(www.sergey.fyi)
419 points serjester | 4 comments | 31 Mar 25 14:45 UTC | HN request time: 0.835s | source
Show context
cryptoz ◴[31 Mar 25 15:03 UTC] No.43535849[source]▶
This is refreshing to read. I, like everyone apparently, am working on my own coding agent [1]. And I suppose it's not that capable yet. But it sure is getting more reliable. I have it only modify 1 file at a time. It generates tickets for itself to complete - but never enough tickets to really get all the work done. The tickets it does generate, however, it often can complete (at least, in simple cases haha). The file modification is done through parsing ASTs and modifying those, so the AI doesn't go off and do all kinds of things to your whole codebase.

And I'm so sick of everything trying for 100% automation and failing. There's a place for the human in the loop, in quickly identifying bugs the AI doesn't have the context for, or large-scale vision, or security or product-focused mindset, etc.

It's going to be AI and humans collaborating. The solutions that figure that out the best are going to win IMO. AI won't be doing everything and humans won't be doing it all either. The tools with the best human-AI collaboration are where it's at.

[1] https://codeplusequalsai.com

replies(1): >>43536048 #
1. helltone ◴[31 Mar 25 15:19 UTC] No.43536048[source]▶
How do you modify ASTs?
replies(1): >>43536084 #
2. cryptoz ◴[31 Mar 25 15:22 UTC] No.43536084[source]▶
I support HTML, JS, Python and CSS. For HTML, (not technically an AST), I give the LLM the original-file HTML source, and then I instruct it to write python code that uses BeautifulSoup to modify the HTML. Then I get the string back from python of the full HTML file, modified according to the user prompt.

For python changes I use ast and astor packages, for JS I use esprima/escodegen/estraverse, and for CSS I use postcss. The process is the same for each one: I give the original input souce file, and I instruct the LLM to parse the file into AST form and then write code that modifies that AST.

I blogged about it here if you want more details! https://codeplusequalsai.com/static/blog/prompting_llms_to_m...

replies(1): >>43537265 #
3. skydhash ◴[31 Mar 25 17:07 UTC] No.43537265[source]▶
I took a look at your project and while it's nice (technically), for the actual use case shown, I can't see the value over something like the old Dreamweaver with a bit of training.

I still think like prompting is still the wrong interface for programming systems. Even though they're restricted, configurations forms, visual programming with nodes, and small scripts attached to objects on a platform is way more reliable and useful.

replies(1): >>43538323 #
4. cryptoz ◴[31 Mar 25 18:46 UTC] No.43538323{3}[source]▶
Appreciate you having a look and for that feedback, thanks - I do agree I have work to do to prove that my idea is better than alternatives. We'll see...