Adding a feature because ChatGPT incorrectly thinks it exists

It’s not that they added a new feature because there was demand.

They added a new feature because technology hallucinated a feature that didn’t exist.

The savior of tech, generative AI, was telling folks a feature existed that didn’t exist.

That’s what the headline is, and in a sane world the folks that run ChatGPT would be falling over themselves to be sure it didn’t happen again, because next time it might not be so benign as it was this time.

This would be a world without generative AI available to the public, at the moment. Requiring perfection would either mean guardrails that would make it useless for most cases, or no LLM access until AGI exists, which are both completely irrational, since many people are finding practical value in its current imperfect state.

The current state of LLM is useful for what it's useful for, warnings of hallucinations are present on every official public interface, and its limitations are quickly understood with any real use.

Nearly everyone in AI research is working on this problem, directly or indirectly.

If “don’t hallucinate” is too much to ask then ethics flew out the window long ago.

> If “don’t hallucinate” is too much to ask then ethics flew out the window long ago.

Those sentences aren't compatible.

> but hallucination is a major issue

Again, every official public AI interface has warnings/disclaimers for this issue. It's well known. It's not some secret. Every AI researcher is directly or indirectly working on this.

> is in the opposite direction of the “goal” of AGI

This isn't a logical statement, so it's difficult to respond to. Hallucination isn't a direction that's being headed towards, it's being actively, with intent and $$$, headed away from.

Really!?

[0] https://i.imgur.com/ly5yk9h.png

What?? What does AGI have to do with this? (If this was some kind of hyperbolic joke, sorry, i didn't get it.)

But, more importantly, the GP only said that in a sane world, the ChatGPT creators should be the ones trying to fix this mistake on ChatGPT. After all, it's obviously a mistake on ChatGPT's part, right?

That was the main point of the GP post. It was not about "requiring perfection" or something like that. So please let's not attack a straw man.

My web browser isn't perfect, but it does not hallucinate inexistent webpages. It sometimes crashes, it sometimes renders wrong, it has bugs and errors. It does not invent plausible-looking information.

There really is a lot middle gound between perfect and "accept anything we give you, no matter how huge the problems".

Their requirement is no hallucinations [1], also stated as "be sure it didn't happen again" in the original comment. If you define a hallucination as something that wasn't in the training data, directly or indirectly (indirectly being something like an "obvious" abstract concept), then you've placed a profound constraint on the system, requiring determinism. That requirement fundamentally, by the non-deterministic statistics that these run on, means you cannot use an LLM, as they exist today. They're not "truth" machines - use a database instead.

Saying "I don't know", with determinism is only slightly different than saying "I know" with determinism, since it requires being fully aware of what you do know, not at a fact level, but at a conceptual/abstract level. Once you have a system that fully reasons about concepts, is self aware of its own knowledge, and can find the fundamental "truth" to answer a question with determinism, you have something indistinguishable from AGI.

Of course, there's a terrible hell that lives between those two, in the form of: "Error: Question outside of known questions." I think a better alternative to this hell would be a breakthrough that allowed "confidence" to be quantified. So, accept that hallucinations will exist, but present uncertainty to the user.

[1] https://news.ycombinator.com/item?id=44496098

> it sometimes renders wrong

Is close to equivalent.

Meanwhile, sensible people have concluded that, even though it isn’t perfect, Wikipedia is still very, very useful – despite the possibility of being misled occasionally.

A technology can be extremely useful despite not being perfect. Failure cases can be taken into consideration rationally without turning it into a moral panic.

You have no ability to edit Wikipedia to stop it from lying. Somebody can come along and re-add the lie a millisecond later.

And yes, I do have the ability to edit Wikipedia. Anyone can edit Wikipedia. I can go on any page I want, right now, and make changes to the page. If someone readds a lie, then eventually we can hit consensus as other editors enter the discussion. Wikipedia's basis is formed by consensus and controlled by individuals like you and I.

ChatGPT is not. It is controlled by one company; I cannot go edit the weights of ChatGPT to prevent it from lying about my app or anything else I do. I can only petition them to change it and hope that either I have enough clout or have a legal basis to do so.

> how do I edit ChatGPT so it stops lying then?

This is what you have changed to:

> And yes, I do have the ability to edit Wikipedia.

You do not have the ability to edit Wikipedia so it stops lying, which is the relevant factor here.

Wikipedia edits are monitored and vandalism is taken seriously, especially on the more important pages.

There is a chasm of difference between being misled occasionally (Wikipedia) and frequently (LLMs). I don’t think you understand how much effort goes on behind the scenes at Wikipedia. No, not everyone can edit every Wikipedia page willy-nilly. Pages for major political figures often can only be edited with an account. IPs like those of iCloud Private Relay are banned and can’t anonymously edit the most basic of pages.

Furthermore, Wikipedia was always honest about what it is from the start. They managed expectations, underpromised and overdelivered. The bozos releasing LLMs talk about them as if they created the embryo of god, and giving money to their religion will solve all your problems.

This is where your analogy is falling apart; of course web browsers do not "invent plausible-looking information" because they don't invent anything in the first place! Web browsers represent a distinct set of capabilities, and as you correctly pointed out, these are often riddled with bugs and errors. If I was making a browser analogy, I would point towards fingerprinting; most browsers reveal too much information about any given user and system, either via cross-site cookies, GPU prints, and whatnot. This is an actual example where "ethics flew out the window long ago."

As the adjacent commenter pointed out: different software, different failure modes.

[1]: https://support.google.com/websearch/answer/14901683

In any case, if you really believe a disclaimer makes it okay for Google to display blatant misinformation in a first-party capacity, we have little to discuss.

Show more -> Disclaimer and the feedback buttons are shown at the end. If you bothered enough to read the full response, you would have seen the disclaimer, but you never did, so you haven't. For something to be considered "misinformation," in the very least the subject of speech has to be asserting its truthfulness—and, indeed, Google makes no such claims. The claim they're making is precisely that its search result-embedded "[..] responses may include mistakes." In this specific case, they are not asserting truthfulness.

FWIW, Gemini 2.5 Pro answers the question correctly.

The search hints are clearly a low-compute first approximation, which is probably correct for most trivial questions which is probably the majority of user queries, and it's not surprising that it fails in this specific instance. The application doesn't allow for reasoning due to scale; even Google cannot afford to run reasoning traces on every search question. I concur that there's very little to discuss: you seemingly made up your mind re: LLM technology, and I doubt you will appreciate the breaking-up of your semantics to begin with.

Ok, sure. But why would you choose to define hallucinations in a way that is contrary to common sense and the normal understanding of what an AI hallucination is?

The common definition of hallucinations is basically: when AI makes shit up and presents it as fact. (And the more technical definition also basically aligns with that.)

No one would say that if the AI takes the data you provide in the prompt and can deduce a correct answer for that specific data —something that is not directly or indirectly present in its training data— it would be hallucinating. In fact that would be an expected thing for an intelligent system to do.

It seems to me you're trying to discuss with something nobody said. You're making it seem that saying "it's bad that LLMs can invent wrong/misleading information like this and present it as fact, and that the companies that deploy them don't seem to care" is equivalent to "i want LLMs to be perfect and have no bugs whatsoever", and then discuss about how ridiculous is to state the latter.

Note that "never being wrong" can also be achieved by an "I looked into it, and there's no clear answer.", which is the correct answer for many questions (humans not required).

> in a sane world the folks that run ChatGPT would be falling over themselves to be sure it didn’t happen again

> If “don’t hallucinate” is too much to ask then ethics flew out the window long ago.

Neither of those ("didn't happen again" and "don't hallucinate") are logically ambiguous or flexible. I can only respond to what they wrote.

I understand Wikipedia puts effort in, but it’s irrelevant. As a user, you can never be sure that what you are reading on Wikipedia is the truth. There are good reasons to assume that certain topics are more safe and certain topics are less safe, but there are no guarantees. The same is true of AI.

> Wikipedia was always honest about what it is from the start.

Every mainstream AI chatbot includes wording like “ChatGPT can make mistakes. Check important info.”

As far as I understand Wikipedia’s rules, no it wouldn’t. Wikipedia considers it an edit war when there are three reverts in a 24 hour period. So if you remove a lie and somebody reverts that change, that is not considered an edit war. If they do it twice, it’s still not an edit war. If they do it three times, but over a greater time period than a single day, it’s still not an edit war. If somebody else re-adds the lie, it’s not an edit war either.

Regardless, it’s not important. The point I am making is that as a user, you cannot be sure that Wikipedia is not serving up lies to you.