←back to thread

925 points dmitrybrant | 3 comments | | HN request time: 0.687s | source
Show context
codedokode ◴[] No.45165982[source]
LLMs are also good for writing quick experiments and benchmarks to satisfy someone's curiosity. For example, once I was wondering, how much time does it take to migrate a cache line between cores when several processes access the same variable - and after I wrote a detailed benchmark algorithm, LLM generated the code instantly. Note that I described the algorithm completely and what it did is just translated it into the code. Obviously I could write the code myself, but I might need to lookup a function (how does one measure elapsed time?), I might make mistakes in C, etc. Another time a made a benchmark to compare linear vs tree search for finding a value in a small array.

It's very useful when you get the answer in several minutes rather than half a hour.

replies(1): >>45166025 #
codedokode ◴[] No.45166025[source]
Also I wanted to add that LLMs (at least free ones) are pretty dumb sometimes and do not notice obvious thing. For example, when writing tests they generate lot of duplicated code and do not move it into a helper function, or do not combine tests using parametrization. I have to do it manually every time.

Maybe it is because they generate the code in one pass and cannot return back and fix the issues. LLM makers, you should allow LLMs to review and edit the generated code.

replies(5): >>45166074 #>>45166289 #>>45166391 #>>45166596 #>>45170444 #
1. scotty79 ◴[] No.45166596[source]
> I have to do it manually every time.

You can tell it to move it and they'll move it and use this shared code from now on.

replies(1): >>45167962 #
2. codedokode ◴[] No.45167962[source]
Sometimes it seems like explaining what I want could take more time than actually editing the code.

For example, imagine if you test a vector-like collection. In every test case dumb LLM creates vector manually and makes inserts/deletes. It could be replaced by adding a helper function that accepts a sequence of operations and returns the processed vector. Furthermore, once you have that function, you can merge multiple tests with parametrization, by having a test function accept a sequence of operation and expected result:

    parametrize('actions, result', (
        # Test that remove removes items from vector
        ([Ins(1, 2, 3, 4), Remove(4)], [1, 2, 3]),
        ...
    )
But it takes time to write this explanation, and dumb LLM might not merge all tests from the first time.
replies(1): >>45168804 #
3. scotty79 ◴[] No.45168804[source]
Try something like:

"Don't create vector manually inline in every test case, make a helper function for that."

and see what agent does. It might do something smart. It might do something a bit dumb but by understanding why exactly it's dumb, you can communicate what correction is needed pretty smoothly.