←back to thread

AI companies cause most of traffic on forums

(pod.geraspora.de)
770 points ta988 | 2 comments | 30 Dec 24 14:37 UTC | HN request time: 0.417s | source
Show context
walterbell ◴[30 Dec 24 16:51 UTC] No.42551009[source]▶
OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper...

For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/

> The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the content on blog post and pages on your site when one of these bots slithers by.

replies(6): >>42551078 #>>42551167 #>>42551217 #>>42551446 #>>42551777 #>>42564313 #
brookst ◴[30 Dec 24 16:58 UTC] No.42551078[source]▶
The latter is clever but unlikely to do any harm. These companies spend a fortune on pre-training efforts and doubtlessly have filters to remove garbage text. There are enough SEO spam pages that just list nonsense words that they would have to.
replies(5): >>42551122 #>>42551337 #>>42551547 #>>42552581 #>>42562028 #
rickyhatespeas ◴[30 Dec 24 17:42 UTC] No.42551547[source]▶
It will do harm to their own site considering it's now un-indexable on platforms used by hundreds of millions and growing. Anyone using this is just guaranteeing that their content will be lost to history at worst, or just inaccessible to most search engines/users at best. Congrats on beating the robots, now every time someone searches for your site they will be taken straight to competitors.
replies(4): >>42551624 #>>42551689 #>>42552139 #>>42553241 #
1. walterbell ◴[30 Dec 24 17:48 UTC] No.42551624[source]▶
> now every time someone searches for your site they will be taken straight to competitors

There are non-LLM forms of distribution, including traditional web search and human word of mouth. For some niche websites, a reduction in LLM-search users could be considered a positive community filter. If LLM scraper bots agree to follow longstanding robots.txt protocols, they can join the community of civilized internet participants.

replies(1): >>42551801 #
2. knuppar ◴[30 Dec 24 18:03 UTC] No.42551801[source]▶
Exactly. Not every website needs to be at the top of SEO (or LLM-O?). Increasingly the niche web feels nicer and nicer as centralized platforms expand.