←back to thread

770 points ta988 | 1 comments | | HN request time: 0.207s | source
Show context
mentalgear ◴[] No.42551541[source]
Note-worthy from the article (as some commentators suggested blocking them).

"If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."

replies(5): >>42551717 #>>42551976 #>>42552122 #>>42552700 #>>42552885 #
loeg ◴[] No.42552122[source]
I'd kind of like to see that claim substantiated a little more. Is it all crawlers that switch to a non-bot UA, or how are they determining it's the same bot? What non-bot UA do they claim?
replies(3): >>42552172 #>>42552177 #>>42555570 #
denschub ◴[] No.42552172[source]
> Is it all crawlers that switch to a non-bot UA

I've observed only one of them do this with high confidence.

> how are they determining it's the same bot?

it's fairly easy to determine that it's the same bot, because as soon as I blocked the "official" one, a bunch of AWS IPs started crawling the same URL patterns - in this case, mediawiki's diff view (`/wiki/index.php?title=[page]&diff=[new-id]&oldid=[old-id]`), that absolutely no bot ever crawled before.

> What non-bot UA do they claim?

Latest Chrome on Windows.

replies(1): >>42553872 #
1. loeg ◴[] No.42553872[source]
Thanks.