←back to thread

493 points todsacerdoti | 2 comments | | HN request time: 0.426s | source
Show context
JonChesterfield ◴[] No.44382974[source]
Interesting. Harder line than the LLVM one found at https://llvm.org/docs/DeveloperPolicy.html#ai-generated-cont...

I'm very old man shouting at clouds about this stuff. I don't want to review code the author doesn't understand and I don't want to merge code neither of us understand.

replies(8): >>44383040 #>>44383128 #>>44383155 #>>44383230 #>>44383315 #>>44383409 #>>44383434 #>>44384226 #
jitl ◴[] No.44383230[source]
When I use LLM for coding tasks, it's like "hey please translate this YAML to structs and extract any repeated patterns to re-used variables". It's possible to do this transform with deterministic tools, but AI will do a fine job in 30s and it's trivial to test the new output is identical to the prompt input.

My high-level work is absolutely impossible to delegate to AI, but AI really helps with tedious or low-stakes incidental tasks. The other day I asked Claude Code to wire up some graphs and outlier analysis for some database benchmark result CSVs. Something conceptually easy, but takes a fair bit of time to figure out libraries and get everything hooked up unless you're already an expert at csv processing.

replies(3): >>44383561 #>>44384921 #>>44386134 #
mattmanser ◴[] No.44384921[source]
In my experience, AI will not do a fine job of things like this.

If the definition is past any sort of length, it will hallucinate new properties, change the names, etc. It also has a propensity to start skipping bits of the definitions by adding in comments like "/** more like this here **/"

It may work for you for small YAML files, but beware doing this for larger ones.

Worst part about all that is that it looks right to begin with because the start of the definitions will be correct, but there will be mistakes and stuff missing.

I've got a PoC hanging around where I did something similar by throwing an OpenAPI spec at an AI and telling it to generate some typescript classes because I was being lazy and couldn't be bothered to run it through a formal tool.

Took me a while to notice a lot of the definitions had subtle bugs, properties were missing and it had made a bunch of stuff up.

replies(2): >>44385398 #>>44387123 #
1. danielbln ◴[] No.44385398[source]
What does "AI" mean? GPT3.5 on a website, or Claude 4 Opus plugged into function calling and a harness of LSP, type checker and tool use? These are not the same, neither in terms of output quality nor in capability space. We need to be more specific about the tools we use when we discuss them. "IDEs are slow to load" wouldn't be a useful statement either.
replies(1): >>44390476 #
2. mattmanser ◴[] No.44390476[source]
How do any of those things help with it recognizing it's hallucinated new property names?

The types don't exist outside of the yaml/json/etc.

You can't check them.