←back to thread

Nepenthes is a tarpit to catch AI web crawlers

(zadzmo.org)
646 points blendergeek | 2 comments | 16 Jan 25 13:57 UTC | HN request time: 0.417s | source
Show context
hartator ◴[16 Jan 25 14:52 UTC] No.42725964[source]▶
There are already “infinite” websites like these on the Internet.

Crawlers (both AI and regular search) have a set number of pages they want to crawl per domain. This number is usually determined by the popularity of the domain.

Unknown websites will get very few crawls per day whereas popular sites millions.

Source: I am the CEO of SerpApi.

replies(9): >>42726093 #>>42726258 #>>42726572 #>>42727553 #>>42727737 #>>42727760 #>>42728210 #>>42728522 #>>42742537 #
1. p0nce ◴[16 Jan 25 17:27 UTC] No.42728210[source]▶
Brand new site with no user gets 1k request a month by bots, the CO2 cost must be atrocious.
replies(1): >>42730712 #
2. tivert ◴[16 Jan 25 20:49 UTC] No.42730712[source]▶
> Brand new site with no user gets 1k request a month by bots, the CO2 cost must be atrocious.

Yep: https://www.energy.gov/articles/doe-releases-new-report-eval...:

> The report finds that data centers consumed about 4.4% of total U.S. electricity in 2023 and are expected to consume approximately 6.7 to 12% of total U.S. electricity by 2028. The report indicates that total data center electricity usage climbed from 58 TWh in 2014 to 176 TWh in 2023 and estimates an increase between 325 to 580 TWh by 2028.

A graph in the report says in data centers used 1.9% in 2018.