Nepenthes is a tarpit to catch AI web crawlers

Question: do these bots not respect robots.txt?

I haven't added these scrapers to my robots.txt on the sites I work on yet because I haven't seen any problems. I would run something like this on my own websites, but I can't see selling my clients on running this on their websites.

The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an IP visits that page, they get added to a blocklist which simply drops their connections without response for 24 hours.

> Question: do these bots not respect robots.txt?

No they don't, because there is no potential legal liability for not respecting that file in most countries.

You haven't seen any problems because you created a solution to the problem!

> The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an IP visits that page, they get added to a blocklist which simply drops their connections without response for 24 hours.

I love this idea!

Yeah, this is elegant as fuck.

Well, I wasn't the original developer who set up every site I work on. Some of the sites I work on don't have this implemented because I wasn't the one who set them up initially.

Even something like a special URL that auto-bans you can be abused by pranksters. Simply embedding an <img> tag that fetches the offending URL could trigger it, as well as tricking people into clicking a link.

Ehhh, is there any reason I should be worried about that? The <img> tag would have to be in a spot where users are likely to go, otherwise users will never view the <img> tag. A link of any kind to the honeypot isn't likely to, for example, go viral on social media, because it's going to appear as a broken link/image and nobody will upvote it. I'm not seeing an attack vector that gets this link in front of my users with enough frequency to be worth considering.

A bigger concern is arguably users who are all behind the same IP address, i.e. some of the sites I work on have employee-only parts which can only be accessed via VPN, so in theory one employee could get the whole company banned, and that would be tricky to figure out. So far that hasn't been a problem, but now that I'm thinking about it, maybe I should have a whitelist override for that. :)

This could be mitigited by having a special secret token in this honeypot URL that limits the time validity of the honeypot url and limits the IP address that this URL is for, let's say: hhtp://example/honeypot/hex(sha256(ipaddress | today(yyyy-mm-dd) | secret))

This special URL with the token would be in an anchor tag somewhere in the footer of every website, but hidden by a CSS rule and "Disallow: /honeypot" rule would be included in robots.txt.