←back to thread

Nepenthes is a tarpit to catch AI web crawlers

(zadzmo.org)
646 points blendergeek | 4 comments | 16 Jan 25 13:57 UTC | HN request time: 0.001s | source
Show context
btbuildem ◴[16 Jan 25 15:46 UTC] No.42726825[source]▶
> ANY SITE THIS SOFTWARE IS APPLIED TO WILL LIKELY DISAPPEAR FROM ALL SEARCH RESULTS

Bug, or feature, this? Could be a way to keep your site public yet unfindable.

replies(1): >>42727107 #
1. chaara-dev ◴[16 Jan 25 16:04 UTC] No.42727107[source]▶
You can already do this with a robots.txt file
replies(3): >>42727297 #>>42727666 #>>42744532 #
2. btbuildem ◴[16 Jan 25 16:17 UTC] No.42727297[source]▶
Technically speaking, yes - but it's in no way enforced, as far as I understand it's more of an honour system.

This malicious solution aligns with incentives (or, disincentives) of the parasitic actors, and might be practically more effective.

3. ◴[16 Jan 25 16:46 UTC] No.42727666[source]▶
4. mqus ◴[18 Jan 25 00:01 UTC] No.42744532[source]▶
> If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex.

> If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result will not have a description.

https://developers.google.com/search/docs/crawling-indexing/...

So, a robots.txt will not keep your site off of google, it just prevents it from getting crawled. (But, to be fair, this tool probably does not do this as well)