←back to thread

AI companies cause most of traffic on forums

(pod.geraspora.de)
770 points ta988 | 2 comments | 30 Dec 24 14:37 UTC | HN request time: 0.001s | source
Show context
Saris ◴[02 Jan 25 14:31 UTC] No.42574712[source]▶
What I don't get is why they need to crawl so aggressively, I have a site with content that doesn't change often (company website) with a few hundred pages total. But the same AI bot will scan the entire site multiple times per day, like somehow all the content is going to suddenly change now after it hasn't for months.

That cannot be an efficient use of their money, maybe they used their own AI to write the scraper code.

replies(1): >>42578182 #
1. hmottestad ◴[02 Jan 25 20:01 UTC] No.42578182[source]▶
The post mentions that the bots were crawling all the wiki diffs. I think that might be useful to see how text evolves and changes over time. Possibly how it improves over time, and what those improvements are.

I guess they are hoping that there will be small changes to your website that it can learn from.

replies(1): >>42579591 #
2. pvaldes ◴[02 Jan 25 22:22 UTC] No.42579591[source]▶
Maybe trying to guess who wrote who?