←back to thread

AI web crawlers are destroying websites in their never-ending content hunger

(www.theregister.com)
211 points CrankyBear | 2 comments | 02 Sep 25 16:24 UTC | HN request time: 0.483s | source
Show context
tmaly ◴[02 Sep 25 17:53 UTC] No.45106642[source]▶
What if content providers reduced the 30k word page for a recipe down to just the actual recipe, would this reduce the amount of data these bots are pulling down?

I don't see this slowing down. If websites don't adapt to the AI deep search reality, the bot will just go somewhere else. People don't want to read these massive long form pages geared at outdated Google SEO techniques.

replies(2): >>45106929 #>>45107135 #
1. gowld ◴[02 Sep 25 18:30 UTC] No.45107135[source]▶
Text content at the sub-page level is approximately 0% of web traffic. It's a non-issue.
replies(1): >>45112650 #
2. immibis ◴[03 Sep 25 06:00 UTC] No.45112650[source]▶
Then remove the 30MB of advertisements, serve just the 3kB of text, and your server load will be completely fine.