Nepenthes is a tarpit to catch AI web crawlers

The arms race between AI bots and bot-protection is only going to get worse, leading to increasing infra costs while negatively impacting the UX and performance (captchas, rate limiting, etc.).

What's a reasonable way forward to deal with more bots than humans on the internet?

It's time to level up in this arms race. Let's stop delivering html documents, use animated rendering of information that is positioned in a scene so that the user has to move elements around for it to be recognizable, like a full site captcha. It doesn't need to be overly complex for the user that can intuitively navigate even a 3D world, but will take x1000 more processing for OpenAI. Feel free to come up with your creative designs to make automation more difficult.

For me, this would finally be a good use case for bitcoin or similar digital transactions. Let the client provide either proof-of-work or proof-of-payment. If we can make the proof of work match the browsing speed of an average human, anything accessing more pages than that will need to provide payment instead.