Has anyone done a talk/blog/whatever on how llm crawlers are different than classical crawlers? I'm not up on the difference.
replies(5):
The trap in the article has a link. Bots are instructed not to follow the link. The link is normally invisible to humans. A client that visits the link is probably therefore a poorly behaved bot.
Actual LLM involvement as the requesting user-agent is vanishingly small. It's the same problem as ever: corporations, their profit motive during $hypecycle coupled with access to capital for IT resources, and the protection of the abusers via the company's abstraction away of legal liability for their behavior.
Back then, legitimate search engines wouldn’t want to scrape things that would just make their search results less relevant with garbage data anyways, so by and large they would honor robots.txt and not overwhelm upstream servers. Bad actors existed, of course, but were very rarely backed by companies valued in the billions of dollars.
People training foundation models now have no such constraints or qualms - they need as many human-written sentences as possible, regardless of the context in which they are extracted. That’s coupled with a broader familiarity with ubiquitous residential proxy providers that can tunnel traffic through consumer connections worldwide. That’s an entirely different social contract, one we are still navigating.
Are these botnets? Are AI companies mass-funding criminal malware companies?
Without a doubt some of them are botnets. AI companies got their initial foothold by violating copyright en masse with pirated textbook dumps for training data, and whatnot. Why should they suddenly develop scruples now?
edit: ah yes another person above mentioned VPN's thats a good possibility, also another vector is users on mobile can sell their extra data that they dont use to 3rd parties. probably many more ways to acquire endpoints.
I wouldn't be surprised if it was the same with LLMs. Millions of workers allocated dynamically on AWS, with varying IPs.
In my specific case, as I was dealing with browser-initiated traffic, I wrote a Firefox add-on instead. No such shortcut for web servers, though.
Your DNS mostly passes lookup requests but during homework time, when there's a request for the ip for "www.youtube.com" it returns the ip of your choice instead of the actual one. The domain's TTL is 5 minutes.
Or don't, technical solutions to social problems are of limited value.
I've deployed the same one for me, but setup for Reddit during work hours.
Both of us know how to get around the add-on. It's not particularly hard. But since Firefox is the primary browser for both of us, it does the trick.
1. Their number: every other company and the mangy mutt that is its mascot is scraping for LLMs at the moment, so you get hit by them far more than you get hit by search engine bots and similar. This makes them harder to block too, because even ignoring tricks like using botnets to spread requests over many source addresses (potentially the residential connections of unwitting users infected by malware) the share number coming from so many places, new places all the time, means you can not maintain a practical blocklist of source addresses. The number of scrapers out there means that small sites can be easily swamped, much like when HN, slashdot, or a popular reddit subsection, links to a site, and it gets “hugged to death” by a sudden glut of individual people who are interested.
2. Use of the information: Search engines actually provide something back: sending people to your site. Useful if that is desirable which in many cases it is. LLMs don't tend to do that though: by the very nature of LLMs very few results from them come with any indication of the source of the data they use for their guesswork. They scrape, they take, they give nothing back. Search engines had a vested interest in your site surviving as they don't want to hand out dead links, those scraping for LLMs have no such requirement because they can still summarise your work from what is effectively cached within their model. This isn't unique to LLMs, go back a few years to the pre-LLM days and you will find several significant legal cases about search engines offering summaries of the information found instead of just sending people to the site where the information is.
3. Ignoring rules: Because so many sites are attempting to block scrapers now, usually at a minimum using accepted methods to discourage it (robots.txt, nofollow attributes, etc.), these signals are just ignored. Sometimes this is malicious with people running the scrapers simply not caring despite knowing the problem they could create, sometimes it is like the spam problem in mail: each scraper thinks it'll be fine because it is only them, with each of the many also thinking the same thing… With people as big as Meta openly defending piracy as just fine for the purposes of LLM training, others see that as a declaration of open season. Those that are malicious or at least amoral (most of them) don't care. Once they have scraped your data they have, as mentioned above, no vested interest in whether your site lives or dies (either by withing away from lack of attention or falling over under their load to never be brought back up), in fact they might have incentive to want your site dead: it would no longer compete with the LLM as a source of information.
No one of these is the problem, but together they are a significant problem.
I'm not affiliated with them, but it has helped me when I really need to focus.
In this case, I don't have a server I can conveniently use as DNS. Plus I wanted to also control the launching of some binaries, so that would considerably complicate the architecture.
Maybe next time :)
Of course, if you don’t care about affecting genuine users then it is much simpler. One could say it’s collateral damage and show a message suggesting to boycott companies and/or business practices that prompted these measures.