Even if we assume there is value in it, why should it replace (even if in part) the previous activity of reliably making computers do exactly what we want?
A contrived example: there are only 100 MB of disk space left, but 1 GB of logs to write. LLM discards 900 MB of logs and keeps only the most important lines.
Sure, you can nitpick this example, but it's the kind of edge case handling that LLMs can "do something resonable" that before required hard coding and special casing.
And it’s not just this specific problem. I don’t think letting an LLM handle edge cases is really ever an appropriate use case in production.
I’d much rather the system just fail so that someone will fix it. Imagine a world where at every level instead of failing and halting, everything error just got bubbled up to an LLM that tried to do something reasonable.
Talk about emergent behavior, or more likely catastrophic cascading failures.
I can kind of see your point if you’re talking about a truly hopeless scenario. Like some imaginary autonomous spacecraft that is going to crash into the sun, so in a last ditch effort the autopilot turns over the controls to an LLM.
But even in that scenario we have to have some way of knowing that we truly are in a hopeless scenario. Maybe it just appears that way and the LLM makes it worse.
Or maybe the LLM decides to pilot it into another spacecraft to reduce velocity.
My point is there aren’t many scenarios where “do something reasonable 90% of the time, but do something insane the other 10% of the time” is better than do nothing.
I’ve been using LLMs at work and my gut feeling saying I’m getting some productivity boost, but I’m not even certain of that because I have also spent time chasing subtle bugs that I wouldn’t have introduced myself. I think I’m going to need to see the results of some large well designed studies and several years of output before I really feel confident saying one way or the other.