←back to thread

Nepenthes is a tarpit to catch AI web crawlers

(zadzmo.org)
646 points blendergeek | 1 comments | 16 Jan 25 13:57 UTC | HN request time: 0.209s | source
Show context
bflesch ◴[16 Jan 25 15:46 UTC] No.42726827[source]▶
Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week.

Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website.

The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As ChatGPT crawler is using various Azure IP ranges I actually think the tarpit would crash first.

The vulnerability reporting experience with OpenAI / BugCrowd was really horrific. It's always difficult to get attention for DOS/DDOS vulnerabilities and companies always act like they are not a problem. But if their system goes dark and the CEO calls then suddenly they accept it as a security vulnerability.

I spent a week trying to reach OpenAI/Microsoft to get this fixed, but I gave up and just published the writeup.

I don't recommend you to exploit this vulnerability due to legal reasons.

[1] https://github.com/bf/security-advisories/blob/main/2025-01-...

replies(8): >>42727288 #>>42727356 #>>42727528 #>>42727530 #>>42733203 #>>42733949 #>>42738239 #>>42742714 #
andai ◴[17 Jan 25 15:04 UTC] No.42738239[source]▶
Is 5000 a lot? I'm out of the loop but I thought c10k was solved decades ago? Or is it about the "burstiness" of it?

(That all the requests come in simultaneously -- probably SSL code would be the bottleneck.)

replies(2): >>42739738 #>>42740516 #
1. bflesch ◴[17 Jan 25 17:10 UTC] No.42740516[source]▶
I'm not a DDOS expert and didn't test out the limits due to potential harm to OpenAI.

Based on my experience I recognized it as potential security risk and framed it as DDOS because there's a big amplification factor: 1 API request via Cloudflare -> 5000 incoming requests from OpenAI

- their requests come in simultaneously from different ips

- each request downloads up to 10mb of random data (tested with multi-gb file)

- the requests come from different azure IP ranges, either bc they kept switching them or bc of different geolocations.

- if you block them on the firewall their requests still hammer your server (it's not like the first request notices it can't establish connection and then the next request TO SAME IP would stop)

I tried to get it recognized and fixed, and now apparently HN did its magic because they've disabled the API :)

Previously, their engineers might have argued that this is a feature and not a bug. But now that they have disabled it, it shows that this clearly isn't intended behavior.