←back to thread

Anubis Works

(xeiaso.net)
319 points evacchi | 6 comments | 12 Apr 25 22:32 UTC | HN request time: 0.94s | source | bottom
Show context
throwaway150 ◴[12 Apr 25 23:06 UTC] No.43668638[source]▶
Looks cool. But please help me understand. What's to stop AI companies from solving the challenge, completing the proof of work and scrape websites anyway?
replies(6): >>43668690 #>>43668774 #>>43668823 #>>43668857 #>>43669150 #>>43670014 #
1. ronsor ◴[13 Apr 25 00:55 UTC] No.43669150[source]▶
I know companies that already solve it.
replies(2): >>43669490 #>>43669768 #
2. creata ◴[13 Apr 25 02:13 UTC] No.43669490[source]▶
Why is spending all that CPU time to scrape the handful of sites that use Anubis worth it to them?
replies(1): >>43670059 #
3. wredcoll ◴[13 Apr 25 03:11 UTC] No.43669768[source]▶
I mean... knowing how to solve it isn't the trick, it's doing it a million times a minute for your firehose scraper.
replies(1): >>43671100 #
4. vhcr ◴[13 Apr 25 04:17 UTC] No.43670059[source]▶
Because it's not a lot of CPU, you only have to solve it once per website, and the default policy difficulty of 16 for bots is worthless because you can just change your user agent so you get a difficulty of 4.
5. udev4096 ◴[13 Apr 25 08:29 UTC] No.43671100[source]▶
Anubis adds a cookie name `within.website-x-cmd-anubis-auth` which can be used by scrapers for not solving it more than once. Just have a fleet of servers whose sole purpose is to extract the cookie after solving the challenges and make sure all of them stay valid. It's not a big deal
replies(1): >>43678196 #
6. fc417fc802 ◴[14 Apr 25 04:58 UTC] No.43678196{3}[source]▶
Requests are associated with the cookie meaning you can trace and block or rate limit as necessary. The cost of solving the PoW is the cost of establishing a new session. If you get blocked you have to solve again.