←back to thread

Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

(composio.dev)
467 points mraniki | 2 comments | 31 Mar 25 12:09 UTC | HN request time: 0.001s | source
Show context
HarHarVeryFunny ◴[31 Mar 25 14:58 UTC] No.43535790[source]▶
I'd like to see an honest attempt by someone to use one of these SOTA models to code an entire non-trivial app. Not a "vibe coding" flappy bird clone or minimal ioS app (call API to count calories in photo), but something real - say 10K LOC type of complexity, using best practices to give the AI all the context and guidance necessary. I'm not expecting the AI to replace the programmer - just to be a useful productivity tool when we move past demos and function writing to tackling real world projects.

It seems to me that where we are today, AI is only useful for coding for very localized tasks, and even there mostly where it's something commonplace and where the user knows enough to guide the AI when it's failing. I'm not at all convinced it's going to get much better until we have models that can actually learn (vs pre-trained) and are motivated to do so.

replies(6): >>43535869 #>>43535969 #>>43536042 #>>43536795 #>>43536842 #>>43538608 #
kaiokendev ◴[31 Mar 25 15:13 UTC] No.43535969[source]▶
I made this NES emulator with Claude last week [0]. I'd say it was a pretty non-trivial task. It involved throwing a lot of NESDev docs, Disch mapper docs, and test rom output + assembly source code to the model to figure out.

[0]: https://kaiokendev.github.io/nes/

replies(2): >>43536216 #>>43539114 #
1. nowittyusername ◴[31 Mar 25 19:53 UTC] No.43539114[source]▶
I am considering training a custom Lora on atari roms and see if i could get a working game out of it with the Loras use. The thinking here is that atari, nes, snes, etc... roms are a lot smaller in size then a program that runs natively on whatever os. Lees lines of code to write for the LLM means less chance of a screw up. take the rom, convert it to assembly, perform very detailed captions on the rom and train.... if this works this would enable anyone to create games with one prompt which are a lot higher quality then the stuff being made now and with less complexity. If you made an emulator with the use of an llm, that means it understands assembly well enough so i think there might be hope for this idea.
replies(1): >>43542230 #
2. kaiokendev ◴[01 Apr 25 02:29 UTC] No.43542230[source]▶
Well the assembly I put into it was written by humans writing assembly intended to be well-understood by anyone reading it. On the contrary, many NES games abuse quirks specific to the NES that you can't translate to any system outside of the NES. Understanding what that assembly code is doing also requires a complete understanding of those quirks, which LLMs don't seem to have yet (My Mapper 4 implementation still has some bugs because my IRQ handling isn't perfect, and many games rely on precise IRQ timing).