←back to thread

Nepenthes is a tarpit to catch AI web crawlers

(zadzmo.org)
646 points blendergeek | 1 comments | 16 Jan 25 13:57 UTC | HN request time: 0.205s | source
Show context
Dwedit ◴[16 Jan 25 18:21 UTC] No.42728867[source]▶
The article claims that using this will "cause your site to disappear from all search results", but the generated pages don't have the traditional "meta" tags that state the intention to block robots.

<meta name="robots" content="noindex, nofollow">

Are any search engines respecting that classic meta tag?

replies(1): >>42730036 #
1. jorams ◴[16 Jan 25 19:52 UTC] No.42730036[source]▶
Yes, all the big search engines respect that meta tag. Some of the big abusive AI crawlers do too, kind of defeating the (stated) point of the tarpit.