Problems the AI industry is not addressing adequately

I can't speak intelligently about how close AGI really is (I do not believe it is but I guess someone somehow somewhere might come up with a brilliant idea that nobody thought of so far and voila).

However I'm flabbergasted by the lack of attention to so-called "hallucinations" (which is a misleading, I mean marketing, term and we should be talking about errors or inaccuracies).

The problem is that we don't really know why LLMs work. I mean you can run the inference and apply the formula and get output from the given input, but you can't "explain" why LLM produced phase A as an output instead of B,C, or N. There's just too many parameters and computations to go though, and the very concept of "explaining" or "understanding" might not even apply here.

And if we can't understand how this thing works, we can't understand why it doesn't work properly (produces wrong output) and also don't know how to fix it.

And instead of talking about it and trying to find a solution everybody moved on to the agents which are basically LLMs that are empowered to perform complex actions IRL.

How does this makes any sense to anybody? I feel like I'm crazy or missing something important.

I get it, a lot of people are making a lot of money and a lot of promises are being made. But this is absolutely fundamental issue that is not that difficult to understand to anybody with a working brain, and yet I am really not seeing any attention paid to it whatsoever.

You can get use out of a hammer without understanding how the strong force works.

You can get use out of an LLM without understanding how every node works.

I guess a counter, is that we don't need to understand how they work to produce a useful output.

They are a magical black box magic 8 ball, that more likely than not gives you the right answer. Maybe people can explain the black box, and make the magic 8 ball more accurate.

But at the end of the day, with a very complex system it will always be some level of black box unreliable magic 8 ball.

So the question then is how do you build an reliable system from unreliable components. Because llms directly are unreliable.

The answer to this is agents, ie feedback loops between multiple llm calls, which in isolation are unreliable, but in aggregate approach reliability.

At the end of the day the bet on agents is a bet that the model companies will not get a model that will magically be 100% correct on the first try.

You can get injured by using a hammer without understanding how it works.

You can damage a company by using a spreadsheet and not understanding how it works.

In your personal opinion, what are the things you should know before using an LLM?

Hammer is not a perfect analogy because of how simple it is, but sure let's go with it.

Imagine that occasionally when getting in contact with the nail it shatters to bits, or goes through the nail as it were liquid, or blows up, or does something else completely unexpected. Wouldn't you want to fix it? And sure, it might require deep understanding of the nature of the materials and forces involved.

That's what I'd do.

LLM hallucinations aren't errors.

LLMs generate text based on weights in a model, and some of it happens to be correct statements about the world. Doesn't mean the rest is generated incorrectly.

THAT. This is what I don't get. Instead of fixing a complex system let's build more complex system based on it knowing that it might not always work.

When you have a complex system that does not always work correctly, you start disassembling it to simpler and simpler components until you find the one - or maybe several - that are not working as designed, you fix whatever you found wrong with them, put the complex system together again, test it to make sure your fix worked, and you're done. That's how I debug complex cloud-based/microservices-infected software systems, that's how they test software/hardware systems found in aircraft/rockets and whatever else. That's such a fundamental principle to me.

If LLM is a black box by definition and there's no way to make it consistently work correctly, what is it good for?..

You know the difference between verification and validation?

You're describing a lack of errors in verification (working as designed/built, equations correct).

GP is describing an error in validation (not doing what we want / require / expect).

> If LLM is a black box by definition and there's no way to make it consistently work correctly, what is it good for?..

many things are unpredictable on the real world. Most of the machines we make are built upon layers of redundancies to make imperfect systems stable and predictable. this is no different.

Use the human brain as an example then. We don't really know how it works. I mean, we know there's neurotransmitters and neural pathways etc (much like nodes in a transformer), but we don't know how exactly intelligence or our thinking process works.

We're also pretty good at working around human 'hallucinations' and other inaccuracies. Whether it be someone having a bad day, a brain fart, or individual clumsiness. eg in a (bad) organisation, sometimes we do it with layers of reviews and committees, much like layers of LLMs judging each other.

I think too much is attached to the notion of "we don't understand how the LLM works". We don't understand how any complicated intelligence works, and potentially won't for the forseeable future.

More generally, a lot of society is built up from empirical understanding of black box systems. I'd claim the field of physics is a prime example. And we've built reliable systems from unreliable components (see the field of distributed systems).

Honestly? Spam and upselling executives on features that don't work. It's a pretty good autocomplete, too.

It is different. Most systems aren't designed to be a slot machine.

Yet RAG systems can perform quite well, so it's a definite proof that you can build something reliable most of the time out of something not reliable in the first place.

A better analogy might be something like medicine. There are many drugs prescribed that are known to help with certain conditions, but their mechanism of action is not known. While there may be research trying to uncover those mechanisms, that doesn't stop or slow down rolling out of the medicine for use. Research goes at its own pace, and very often cannot be sped up by throwing money at it, while the market dictates adoption. I see the same with LLMs. I'm sure this has attracted the attention of more researchers than anything else in this field, but I would expect any progress to be relatively slow.