←back to thread

Nepenthes is a tarpit to catch AI web crawlers

(zadzmo.org)
646 points blendergeek | 2 comments | 16 Jan 25 13:57 UTC | HN request time: 0.001s | source
Show context
bflesch ◴[16 Jan 25 15:46 UTC] No.42726827[source]▶
Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week.

Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website.

The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As ChatGPT crawler is using various Azure IP ranges I actually think the tarpit would crash first.

The vulnerability reporting experience with OpenAI / BugCrowd was really horrific. It's always difficult to get attention for DOS/DDOS vulnerabilities and companies always act like they are not a problem. But if their system goes dark and the CEO calls then suddenly they accept it as a security vulnerability.

I spent a week trying to reach OpenAI/Microsoft to get this fixed, but I gave up and just published the writeup.

I don't recommend you to exploit this vulnerability due to legal reasons.

[1] https://github.com/bf/security-advisories/blob/main/2025-01-...

replies(8): >>42727288 #>>42727356 #>>42727528 #>>42727530 #>>42733203 #>>42733949 #>>42738239 #>>42742714 #
JohnMakin ◴[16 Jan 25 16:16 UTC] No.42727288[source]▶
Nice find, I think one of my sites actually got recently hit by something like this. And yea, this kind of thing should be trivially preventable if they cared at all.
replies(2): >>42727906 #>>42731618 #
dewey ◴[16 Jan 25 17:06 UTC] No.42727906[source]▶
> And yea, this kind of thing should be trivially preventable if they cared at all.

Most of the time when someone says something is "trivial" without knowing anything about the internals, it's never trivial.

As someone working close to the b2c side of a business, I can’t count the amount of times I've heard that something should be trivial while it's something we've thought about for years.

replies(4): >>42728034 #>>42728078 #>>42728234 #>>42729816 #
bflesch ◴[16 Jan 25 17:17 UTC] No.42728078[source]▶
The technical flaws are quite trivial to spot, if you have the relevant experience:

- urls[] parameter has no size limit

- urls[] parameter is not deduplicated (but their cache is deduplicating, so this security control was there at some point but is ineffective now)

- their requests to same website / DNS / victim IP address rotate through all available Azure IPs, which gives them risk of being blocked by other hosters. They should come from the same IP address. I noticed them changing to other Azure IP ranges several times, most likely because they got blocked/rate limited by Hetzner or other counterparties from which I was playing around with this vulnerabilities.

But if their team is too limited to recognize security risks, there is nothing one can do. Maybe they were occupied last week with the office gossip around the sexual assault lawsuit against Sam Altman. Maybe they still had holidays or there was another, higher-risk security vulnerability.

Having interacted with several bug bounties in the past, it feels OpenAI is not very mature in that regard. Also why do they choose BugCrowd when HackerOne is much better in my experience.

replies(1): >>42728271 #
1. fc417fc802 ◴[16 Jan 25 17:34 UTC] No.42728271[source]▶
> rotate through all available Azure IPs, ... They should come from the same IP address.

I would guess that this is intentional, intended to prevent IP level blocks from being effective. That way blocking them means blocking all of Azure. Too much collateral damage to be worth it.

replies(1): >>42737651 #
2. jackcviers3 ◴[17 Jan 25 14:10 UTC] No.42737651[source]▶
It is. There are scraping third party services you can pay for that will do all of this for you, and getting blocked by IP. You then make your request to the third-party scraper, receive the contents, and do with them whatever you need to do.