Most active commenters

    ←back to thread

    Using LLMs at Oxide

    (rfd.shared.oxide.computer)
    694 points steveklabnik | 15 comments | 07 Dec 25 01:17 UTC | HN request time: 1.192s | source | bottom
    Show context
    mcqueenjordan ◴[07 Dec 25 02:22 UTC] No.46178624[source]▶
    As usual with Oxide's RFDs, I found myself vigorously head-nodding while reading. Somewhat rarely, I found a part that I found myself disagreeing with:

    > Unlike prose, however (which really should be handed in a polished form to an LLM to maximize the LLM’s efficacy), LLMs can be quite effective writing code de novo.

    Don't the same arguments against using LLMs to write one's prose also apply to code? Was this structure of the code and ideas within the engineers'? Or was it from the LLM? And so on.

    Before I'm misunderstood as a LLM minimalist, I want to say that I think they're incredibly good at solving for the blank page syndrome -- just getting a starting point on the page is useful. But I think that the code you actually want to ship is so far from what LLMs write, that I think of it more as a crutch for blank page syndrome than "they're good at writing code de novo".

    I'm open to being wrong and want to hear any discussion on the matter. My worry is that this is another one of the "illusion of progress" traps, similar to the one that currently fools people with the prose side of things.

    replies(9): >>46178640 #>>46178642 #>>46178818 #>>46179080 #>>46179150 #>>46179217 #>>46179552 #>>46180049 #>>46180734 #
    1. lukasb ◴[07 Dec 25 02:25 UTC] No.46178642[source]▶
    One difference is that clichéd prose is bad and clichéd code is generally good.
    replies(1): >>46178649 #
    2. joshka ◴[07 Dec 25 02:29 UTC] No.46178649[source]▶
    Depends on what your prose is for. If it's for documentation, then prose which matches the expected tone and form of other similar docs would be clichéd in this perspective. I think this is a really good use of LLMs - making docs consistent across a large library / codebase.
    replies(3): >>46178656 #>>46178665 #>>46178668 #
    3. minimaxir ◴[07 Dec 25 02:30 UTC] No.46178656[source]▶
    I have been testing agentic coding with Claude 4.5 Opus and the problem is that it's too good at documentation and test cases. It's thorough in a way that it goes out of scope, so I have to edit it down to increase the signal-to-noise.
    replies(2): >>46178944 #>>46179605 #
    4. dcre ◴[07 Dec 25 02:33 UTC] No.46178665[source]▶
    Docs also often don’t have anyone’s name on them, in which case they’re already attributed to an unknown composite author.
    5. danenania ◴[07 Dec 25 02:33 UTC] No.46178668[source]▶
    A problem I’ve found with LLMs for docs is that they are like ten times too wordy. They want to document every path and edge case rather focusing on what really matters.

    It can be addressed with prompting, but you have to fight this constantly.

    replies(2): >>46178826 #>>46181865 #
    6. bigiain ◴[07 Dec 25 03:12 UTC] No.46178826{3}[source]▶
    I think probably my most common prompt is "Make it shorter. No more than ($x) (words|sentences|paragraphs)."
    replies(1): >>46181980 #
    7. girvo ◴[07 Dec 25 03:38 UTC] No.46178944{3}[source]▶
    The “change capture”/straight jacket style tests LLMs like to output drive me nuts. But humans write those all the time too so I shouldn’t be that surprised either!
    replies(1): >>46180074 #
    8. diamond559 ◴[07 Dec 25 06:24 UTC] No.46179605{3}[source]▶
    If the goal is to document the code and it gets sidetracked and focuses on only certain parts it failed the test. It just further proves llm's are incapable of grasping meaning and context.
    9. mulmboy ◴[07 Dec 25 08:09 UTC] No.46180074{4}[source]▶
    What do these look like?
    replies(1): >>46180900 #
    10. pmg101 ◴[07 Dec 25 11:14 UTC] No.46180900{5}[source]▶

      1. Take every single function, even private ones.
  2. Mock every argument and collaborator.
  3. Call the function.
  4. Assert the mocks were  called in the expected way.
    These tests help you find inadvertent changes, yes, but they also create constant noise about changes you intend.
    replies(3): >>46182913 #>>46183812 #>>46185551 #
    11. pxc ◴[07 Dec 25 14:24 UTC] No.46181865{3}[source]▶
    > A problem I’ve found with LLMs for docs is that they are like ten times too wordy

    This is one of the problems I feel with LLM-generated code, as well. It's almost always between 5x and long and 20x (!) as long as it needs to be. Though in the case of code verbosity, it's usually not because of thoroughness so much as extremely bad style.

    12. pxc ◴[07 Dec 25 14:39 UTC] No.46181980{4}[source]▶
    I've never been able to get that to work. LLMs can't count; they don't actually know how long their output is.
    13. ornornor ◴[07 Dec 25 16:31 UTC] No.46182913{6}[source]▶
    Juniors on one of the teams I work with only write this kind of tests. It’s tiring, and I have to tell them to test the behaviour, not the implementation. And yet every time they do the same thing. Or rather their AI IDE spits these out.
    14. senbrow ◴[07 Dec 25 18:25 UTC] No.46183812{6}[source]▶
    These tests also break encapsulation in many cases because they're not testing the interface contract, they're testing the implementation.
    15. girvo ◴[07 Dec 25 21:53 UTC] No.46185551{6}[source]▶
    You beat me to it, and yep these are exactly it.

    “Mock the world then test your mocks”, I’m simply not convinced these have any value at all after my nearly two decades of doing this professionally