Nepenthes is a tarpit to catch AI web crawlers

Bot detection is fairly sophisticated these days. No one bypasses it by accident. If they are getting around it then they are doing it intentionally (and probably dedicating a lot of resources to it). I'm pro-scraping when bots are well behaved but the circumvention of bot detection seems like a gray-ish area.

And, yes, I know about Facebook training on copyrighted books so I don't put it above these companies. I've just never seen it confirmed that they actually do it.

If you enable Cloudflare Captcha, you'll see basically no more bots, only the most persistent remain (that have an active interest in you/your content and aren't just drive-by-hits).

It's just that having the brief interception hurts your conversion rate. Might depend on industry, but we saw 20-30% drops in page views and conversions which just makes it a nuclear option when you're under attack, but not something to use just to block annoyances.

Why do you attribute this to only the "brief interception"? Shouldn't the logical conclusion be that Cloudflare may block 20-30% of regular traffic?