o Claude goes away for 15 minutes, doesn't profile anything, many code changes.
o Announces project now performs much better, saving 70% CPU.
- Claude, test the performance.
o Performance is 1% _slower_ than previous.
- Claude, can I have a refund for the $15 you just wasted?
o [Claude waffles], "no".
Any "performance" axis could have been used: Number of db hits, memory pressure, cpu usage, whatever.
The LLM chose (or whatever) to use CPU performance, claimed a specific figure, and that figure was demonstrably not real.
If you ask a barista to make you a better coffee, and the barista says "this coffee is hotter" and it just isn't, the problem is not underspecified requirements, the problem is that it just doesn't make any attempt to say things that are only correct. Technically it can't make any attempt.
If I tell an intern "Optimize this app for performance" and they come back having reduced the memory footprint by half, but that didn't actually matter because the app was never memory constrained, I could hem and haw about not giving clear instructions, but I could also use that as a teachable moment to help the budding engineer learn how to figure out what matters when given that kind of leeway, to still have impact.
If they instead come back and say "I cut memory usage in half" and then you have them run the app and it has the exact same memory usage, you don't think about not giving clear enough instructions, because you should be asking the intern "Why are you lying to my face?" and "Why are you confidently telling me something you did not verify?".