←back to thread

Nepenthes is a tarpit to catch AI web crawlers

(zadzmo.org)
646 points blendergeek | 1 comments | 16 Jan 25 13:57 UTC | HN request time: 0.247s | source
Show context
bflesch ◴[16 Jan 25 15:46 UTC] No.42726827[source]▶
Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week.

Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website.

The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As ChatGPT crawler is using various Azure IP ranges I actually think the tarpit would crash first.

The vulnerability reporting experience with OpenAI / BugCrowd was really horrific. It's always difficult to get attention for DOS/DDOS vulnerabilities and companies always act like they are not a problem. But if their system goes dark and the CEO calls then suddenly they accept it as a security vulnerability.

I spent a week trying to reach OpenAI/Microsoft to get this fixed, but I gave up and just published the writeup.

I don't recommend you to exploit this vulnerability due to legal reasons.

[1] https://github.com/bf/security-advisories/blob/main/2025-01-...

replies(8): >>42727288 #>>42727356 #>>42727528 #>>42727530 #>>42733203 #>>42733949 #>>42738239 #>>42742714 #
JohnMakin ◴[16 Jan 25 16:16 UTC] No.42727288[source]▶
Nice find, I think one of my sites actually got recently hit by something like this. And yea, this kind of thing should be trivially preventable if they cared at all.
replies(2): >>42727906 #>>42731618 #
dewey ◴[16 Jan 25 17:06 UTC] No.42727906[source]▶
> And yea, this kind of thing should be trivially preventable if they cared at all.

Most of the time when someone says something is "trivial" without knowing anything about the internals, it's never trivial.

As someone working close to the b2c side of a business, I can’t count the amount of times I've heard that something should be trivial while it's something we've thought about for years.

replies(4): >>42728034 #>>42728078 #>>42728234 #>>42729816 #
grahamj ◴[16 Jan 25 17:14 UTC] No.42728034[source]▶
If you’re unable to throttle your own outgoing requests you shouldn’t be making any
replies(1): >>42728110 #
bflesch ◴[16 Jan 25 17:20 UTC] No.42728110[source]▶
I assume it'll be hard for them to notice because it's all coming from Azure IP ranges. OpenAI has very big credit card behind this Azure account so this vulnerability might only be limited by Azure capacity.

I noticed they switched their crawler to new IP ranges several times, but unfortunately Microsoft CERT / Azure security team didn't answer to my reports.

If this vulnerability is exploited, it hits your server with MANY requests per second, right from the hearts of Azure cloud.

replies(1): >>42728152 #
grahamj ◴[16 Jan 25 17:23 UTC] No.42728152[source]▶
Note I said outgoing, as in the crawlers should be throttling themselves
replies(1): >>42728310 #
bflesch ◴[16 Jan 25 17:37 UTC] No.42728310[source]▶
Sorry for misunderstanding your point.

I agree it should be throttled. Maybe they don't need to throttle because they don't care about cost.

Funny thing is that servers from AWS were trying to connect to my system when I played around with this - I assume OpenAI has not moved away from AWS yet.

Also many different security scanners hitting my IP after every burst of incoming requests from the ChatGPT crawler Azure IP ranges. Quite interesting to see that there are some proper network admins out there.

replies(2): >>42729758 #>>42729871 #
1. jillyboel ◴[16 Jan 25 19:38 UTC] No.42729871[source]▶
They need to throttle because otherwise they're simply a DDoS service. It's clear they don't give a fuck though, like any bigtech company. They'll spend millions on prosecuting anyone who dares to do what they perceive as a DoS attack against them, but they'll spit in your face and laugh at you if you even dare to claim they are DDoSing you.