←back to thread

Google AI Overview made up an elaborate story about me

(bsky.app)
693 points jsheard | 10 comments | 01 Sep 25 14:27 UTC | HN request time: 0.001s | source | bottom
Show context
slightwinder ◴[01 Sep 25 15:09 UTC] No.45093284[source]▶
Searching for "benn jordan isreal", the first result for me is a video[0] from a different creator, with the exact same title and date. There is no mentioning of "benn" in the video, but some mentioning of jordan (the country). So maybe, this was enough for Google to hallucinate some connection. Highly concerning!

[0] https://www.youtube.com/watch?v=qgUzVZiint0

replies(3): >>45093342 #>>45093749 #>>45095962 #
trjordan ◴[01 Sep 25 15:53 UTC] No.45093749[source]▶
This is almost certainly what happened. Google's AI answers aren't magic -- they're just summarizing across searches. In this case, "Israel" + "Jordan" pulled back a video with opposite views than the author.

It's somewhat less obvious to debug, because it'll pull more context than Google wants to show in the UI. You can see this happening in AI mode, where it'll fire half a dozen searches and aggregate snippets of 100+ sites before writing its summary.

replies(3): >>45094262 #>>45094296 #>>45095306 #
sigmoid10 ◴[01 Sep 25 18:29 UTC] No.45095306[source]▶
There is actually a musician called Benn Jordan who was impersonated by someone on twitter who posted pro-Israel content [1]. That content is no longer available, but it might have snuck into the training data, i.e. Benn Jordan = pro Israel. This might also have been set in relation to the other Jordan's previous pro-Palestine comments, eventually misattributing the "I was wrong about Israel" video. It's still a clear fuckup - but I could see humans doing something similar when sloppily accruing information.

[1] https://www.webpronews.com/musician-benn-jordan-exposes-fake...

replies(1): >>45095621 #
ants_everywhere ◴[01 Sep 25 19:06 UTC] No.45095621[source]▶
That article is about the same Benn Jordan in the Bluesky post. The photo in the article is not of Benn Jordan.

Benn Jordan has several videos and projects devoted to "digital sabotage", e.g. https://www.google.com/search?hl=en&q=benn%20jordan%20data%2...

So this all kind of looks on its face like it's just him trolling. There may be ore than just what's on the face of course. For example, it could be someone else trolling him with his own methods.

replies(1): >>45095667 #
1. sigmoid10 ◴[01 Sep 25 19:12 UTC] No.45095667[source]▶
That makes it even more believable that an LLM screwed up. I mean what are you supposed to believe at this point?
replies(1): >>45095711 #
2. ants_everywhere ◴[01 Sep 25 19:18 UTC] No.45095711[source]▶
I guess so.

But the situation we're in is that someone who does misinformation is claiming an LLM believed misinformation. Step one would be getting an someone independent, ideally with some journalistic integrity, to verify Benn's claims.

Generally speaking if your aunt sally claims she ate strawberry cake for her birthday, the LLM or Google search has no way of verifying that. If Aunt Sally uploads a faked picture of her eating strawberry cake, the LLM is not going to go to her house and try to find out the truth.

So if Aunt Sally is lying about eating strawberry cake, it's not clear what search is supposed to return when you ask whether she ate strawberry cake.

replies(2): >>45096952 #>>45101414 #
3. lazide ◴[01 Sep 25 21:54 UTC] No.45096952[source]▶
Eventually people are just going to think everything except what they want to believe is a lie. Oh wait, that’s where we are right now.

Good thing I know aunt Sally is a pathological liar and strawberry cake addict, and anyone who says otherwise is a big fat fake.

replies(1): >>45097354 #
4. ants_everywhere ◴[01 Sep 25 22:58 UTC] No.45097354{3}[source]▶
I doubt it. But you can't simultaneously cultivate an image as a propagandist who lies to and about AI and a truth teller who tells the truth about AI.

You either try hard to tell the objective truth or you bend the truth routinely to try to make a "larger" point. The more you do the latter the less credit people will give your word.

replies(1): >>45097376 #
5. lazide ◴[01 Sep 25 23:01 UTC] No.45097376{4}[source]▶
Who is ‘you’ kemosabe?
replies(2): >>45097658 #>>45097668 #
6. ◴[01 Sep 25 23:49 UTC] No.45097658{5}[source]▶
7. ants_everywhere ◴[01 Sep 25 23:51 UTC] No.45097668{5}[source]▶
One, as in, one can be sarcastic and dismissive or one can contribute to the discussion but not both
replies(1): >>45097902 #
8. lazide ◴[02 Sep 25 00:31 UTC] No.45097902{6}[source]▶
And which do you think you’re being?
replies(1): >>45098220 #
9. ◴[02 Sep 25 01:34 UTC] No.45098220{7}[source]▶
10. sigmoid10 ◴[02 Sep 25 11:05 UTC] No.45101414[source]▶
>Step one would be getting an someone independent, ideally with some journalistic integrity

That's already part of the problem. Who defines what integrity is? How do you measure it? And even if you come up with something, how do you convince everyone to agree on it? One person's most trusted source will always be just another bought spindoctor to the next. I don't think this problem is salvageable anymore. I think we need to consider the possibility that the internet will die as a source for any objective information.