Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

Try asking it to generate a high-level plan of how it's going to do the conversion first, then to generate function definitions for the new functions, then have it generate tests for the new functions, then actually write them, while giving it the output of the tests.

It's not like people just one-shot a whole module of code, why would LLMs?

I know many people who can and will one-shot a rewrite of 500 LOC. In my world, 500 LOC is about the length of a single function. I don't understand why we should be talking about generating a high level plan with multiple tests etc. for a single function.

And I don't think this is uncommon. Just a random example from Github, this file is 1800 LOC and 4 functions. It implements one very specific thing that's part of a broader library. (I have no affiliation with this code.)

https://github.com/elemental/Elemental/blob/master/src/optim...

> I don't understand why we should be talking about generating a high level plan with multiple tests etc. for a single function.

You don't have to, you can write it by hand. I thought we were talking about how we can make computers write code, instead of humans, but it seems that we're trying to prove that LLMs aren't useful instead.

> It's not like people just one-shot a whole module of code, why would LLMs?

For conversions between languages or libraries, you often do just one-shot it, writing or modifying code from start to end in order.

I remember 15 years ago taking a 10,000 line Java code base and porting it to JavaScript mostly like this, with only a few areas requiring a bit more involved and non-sequential editing.

Only 500 lines? That's miniscule.

No, it's simply being demonstrated that they're not as useful as some claim.

By saying "why do I have to use a specific technique, instead of naively, to get what I want"?

"Why do I have to put in more work to use this tool vs. not using it?"

If we have to break the problem into tiny pieces that can be individually tested in order for LLMs to be useful, I think it clearly limits LLM usability to a particular niche of programming.

Which is exactly what I said here:

https://news.ycombinator.com/item?id=43537443

You don't have to, the LLM will.

So, you didn't test it until the end? or did you have to build it in such a way that is was partially testable?

> If we have to break the problem into tiny pieces that can be individually tested

Isn't this something that we should have doing for decades of our own volition?

Separation of concerns, single responsibility principle, all of that talk and trend of TDD or at the very least having good test coverage, or writing code that at least can be debugged without going insane (no Heisenbugs, maybe some intermediate variables to stop on in a debugger, instead of just endless chained streams, though opinions are split, at least code that is readable and not 3 pages worth per function).

Because when I see long bits of code that I have to change without breaking anything surrounding them, I don't feel confident in doing that even if it's a codebase I'm familiar with, much less trust an AI on it (at that point it might be a "Hail Mary", a last ditch effort in hoping that at least the AI can find method in the madness before I have to get my own hands dirty and make my hair more gray).

One of the nifty things about the target being JavaScript was that I didn’t have to finish it before I could run it—it was the sort of big library where typical code wouldn’t use most of the functionality. It was audio stuff, so there were a couple of core files that needed more careful porting (from whatever in Java to Mozilla’s Audio Data API, which was a fairly good match), and then the rest was fairly routine that could be done gradually, as I needed them or just when I didn’t have anything better to focus on. Honestly, one of the biggest problems was forgetting to prefix instance properties with `this.`

I think this shows how the approach LLMs take is wrong. For us it's easy because we simply sort of iterate over every function with a simple prompt of doing a translation, but are yet careful enough taking notes of whatever may be relevant to do a higher level change if necessary.

Maybe the mistake is mistaking LLMs as capable people instead of a simple, but optimised neuron soup tuned for text.