Anubis Works

(xeiaso.net)

313 points evacchi | 2 comments | 12 Apr 25 22:32 UTC | HN request time: 0.44s | source

Show context

prologic ◴[13 Apr 25 00:47 UTC] No.43669126[source]▶

I've read about Anubis, cool project! Unfortunately, as pointed out in the comments, requires your site's visitors to have Javascript™ enabled. This is totally fine for sites that require Javascript™ anyway to enhance the user experience, but not so great for static sites and such that require no JS at all.

I built my own solution that effectively blocks these "Bad Bots" at the network level. I effectively block the entirety of several large "Big Tech / Big LLM" networks entirely at the ASN (BGP) by utilizing MaxMind's database and a custom WAF and Reverse Proxy I put together.

replies(4): >>43669138 #>>43669313 #>>43669553 #>>43670788 #

1. jadbox ◴[13 Apr 25 00:52 UTC] No.43669138[source]▶

>>43669126 #

How do you know it's an LLM and not a VPN? How do you use this MaxMind's database to isolate LLMs?

replies(1): >>43669162 #

2. prologic ◴[13 Apr 25 00:58 UTC] No.43669162[source]▶

>>43669138 (TP) #

I don't distinguish actually. There are two things I do normally:

- Block Bad Bots. There's a simple text file called `bad_bots.txt` - Block Bad ASNs. There's a simple text file called `bad_asns.txt`

There's also another for blocking IP(s) and IP-ranges called `bad_ips.txt` but it's often more effective to block an much larger range of IPs (At the ASN level).

To give you an concrete idea, here's some examples:

$ cat etc/caddy/waf/bad_asns.txt # CHINANET-BACKBONE No.31,Jin-rong Street, CN # Why: DDoS 4134

# CHINA169-BACKBONE CHINA UNICOM China169 Backbone, CN # Why: DDoS 4837

# CHINAMOBILE-CN China Mobile Communications Group Co., Ltd., CN # Why: DDoS 9808

# FACEBOOK, US # Why: Bad Bots 32934

# Alibaba, CN # Why: Bad Bots 45102

# Why: Bad Bots 28573

↑