ChatGPT creates phisher's paradise by serving the wrong URLs for major companies

It's wild how many of the links are hallucinations.

Maybe the error rate is consistent with everything else, we just eisegesis our way into thinking it's not

This is, in turn, making the world of comment and forum spam much worse. Site operators could tag all user-submitted links as "nofollow," making their sites useless for SEO spammers. But spammers have learned that most LLM content scraper bots don't care about "nofollow," so they're back to spamming everywhere.

I've been using Phind lately and I think they do a really good job avoiding this problem. I don't think I've ever run into a fake URL using it. As a search engine, I think I still prefer Kagi but Phind is a great if you want a free option.

The study they link to: https://www.netcraft.com/blog/large-language-models-are-fall...

It seems they are selling some product to protect against this. So, I could believe the headline here, but less confident that this is an unbiased look at the problem.

> "It's actually quite similar to some of the supply chain attacks we've seen before [...] you're trying to trick somebody who's doing some vibe coding into using the wrong API."

I have renewed faith in the universe. It totally makes sense that vibe coding would be poisoned into useless oblivion so early in this game.

I wonder if Cloudflare's new plan for blocking AI from scraping the "real" sites...

You wonder if the plan is (or does) what?

I’m not sure if even for traditional search engines “nofollow” means that the scraper doesn’t follow the link, or that it just does not include it in the PageRank or whatever graph but still uses it for to discover new pages. (Of course, LLMs are far too impenetrable for such a middle ground to exist.)

LLMs dont seem to hallucinate my niche products and company (sometimes the name of the company doing the product yes, but not the product name, not the url of the company), and according to CloudFlare Radar I'm only between 200000 and 500000 top domains https://radar.cloudflare.com/scan

A cynic. I like it.

It reminds me of non-radioactive steel, the kind you can only get from ships sunk before the atomic bomb. Someday, we’ll be scavenging for clean data the same way: pre-AI, uncontaminated by the AI explosion of junk.

I feel a bit icky, but whenever I see people work with such a disregard for quality, I'm actually rooting for their products to break and get hacked.

In 2015 it was copy and pasting code from stackoverflow, in 2020 it was npm install left-pad, in 2025 it's vibecoding.

I refuse to join them, and I patiently await the day of rapture.

> “Crims have cottoned on to a new way to lead you astray”

Was - was the article written by AI?

I confess to vibe coding. Specially with api work. And it has gotten so badly that I have to actually either send api pdfs and links to api documentation.

It may already be here but in large moneyed organizations no one wants to take responsibility and speaking against management’s orders to lean in on AI may be a political suicide.

So a lot of people are quietly watching lots of ships slowly taking water.

I’ve had Claude hallucinate options for a lot of things, most recently DOM method arguments, and Wrangler configuration. It’s like, very reasonable things you might expect to exist. But they don’t.

I must be holding it wrong. How are people working with these tools to produce quality software?

In case you're not being sarcastic: they're not.

No, it just sounds British.

Australian. "Crims" isnt said much in the UK, for us that's "thugs" or "yobs".

leftpad was 2016

Not just URL's. Google's AI snippet at the top of a search result page recently fed me the wrong phone number for Starlink support. It connected me to a scam being run at scale out of an overseas call center.

After questions like "Is your router plugged in", "What lights are on", etc., they eventually asked for my account email address and password (I was already getting suspicious by that point and of course did not provide). I could hear other agents in the background having similar conversations with other callers.

Based on some probing questions I asked, I think the agents themselves didn't even realize they weren't really working for Starlink.

I was in a hurry and should have double checked the number before calling, rather than blindly trusting the AI shoveled slop.

I tried but wasn't able to recreate the search query that produced the result, and a Google search for the number itself came up blank. A few weeks later, the number just rings with no answer.