Some thoughts on LLMs and software development

As Fowler himself states, there's a need to learn to use these tools properly.

In any case poor work quality is a failure of tech leadership and culture, it's not AI's fault.

It’s funny how nothing seems to be AI’s fault.

No one seems to be able to grasp the possibility that AI is a failure

If poor work gets merged, the responsibility lies in who wrote it, who merged it, and who allows such a culture.

The tools used do not hold responsibilities, they are tools.

How could a tool be at fault? If an airplane crashes is the plane at fault or the designers, engineers, and/or pilot?

> No one seems to be able to grasp the possibility that AI is a failure.

Do you think by the time GPT-9 comes, we'll say "That's it, AI is a failure, we'll just stop using it!"

Or do you speak in metaphorical/bigger picture/"butlerian jihad" terms?

You've failed to figure out when and how to use it. It's not a binary failed/succeeded thing.

None of the copyright issues or suicide cases are handled in the court yet. There are many aspects.

Metaverse was...

That's because it's software / an application. I don't blame my editor for broken code either. You can't put blame on software itself, it just does what it's programmed to do.

But also, blameless culture is IMO important in software development. If a bug ends up in production, whose fault is it? The developer that wrote the code? The LLM that generated it? The reviewer that approved it? The product owner that decided a feature should be built? The tester that missed the bug? The engineering organization that has a gap in their CI?

As with the Therac-25 incident, it's never one cause: https://news.ycombinator.com/item?id=45036294

Blameless culture is important for a lot of reasons, but many of them are human. LLMs are just tools. If one of the issues identified in a post-mortem is "using this particular tool is causing us problems", there's not a blameless culture out there that would say "We can't blame the tool..."; the action item is "Figure out how to improve/replace/remove the tool so it no longer contributes to problems."

I don't see the use-case now, maybe there will be one by GPT-9

Designers, engineers, and/or pilots aren't tools, so that's a strange rhetorical question.

At any rate, it depends on the crash. The NTSB will investigate and release findings that very well may assign fault to the design of the plane and/or pilot or even tools the pilot was using, and will make recommendations about how to avoid a similar crash in the future, which could include discontinuing the use of certain tools.

“There’s no use for this thing!” - said the farmer about the computer

"I got rid of that machine saw. Every so often it made a cut that was slightly off line but it was hard to see. I might not find out until much later and then have to redo everything."

Absence of your need isn't evidence of no need.

If your toaster burns your breakfast bread, Do you ultimately blame "it"?

You gdt mad, swear at it, maybe even throw it to the wall on a git of rage but, at the end of the day, deep inside you still know you screwed.

Devices can be faulty and technology can be inappropriate.

If I bought an AI powered toaster that allows me to select a desired shade of toast, I select light golden brown, and it burns my toast, I certainly do blame “it”.

I wouldn’t throw it against a wall because I’m not a psychopath, but I would demand my money back.

> You can't put blame on software itself, it just does what it's programmed to do.

This isn't what AI enthusiasts say about AI though, they only bring that up when they get defensive but then go around and say it will totally replace software engineers and is not just a tool.

This is true, but I've never heard of a use case. To which you might reply, "doesn't mean there isn't one," which you would be also right about.

Maybe you know one.

I presume your definition of use case is something that doesn't include what people normally use it for. And I presume me using it for coding every day is disqualified as well.

Blame is purely social and purely human. “Blaming” a tool or process and root causing are functionally identical. Misattributing an outage to a single failure is certainly one way to fail to fix a process. Failing to identify faulty tools/ faulty applications is another way.

I was being flippant to say it’s never AI’s fault, but due to board/C-Suite pressure it’s harder than ever to point out the ways that AI makes processes more complex, harder to reason about, stochastic, and expensive. So we end up with problems that have to be attributed to something not AI.