Anubis Works

(xeiaso.net)

313 points evacchi | 4 comments | 12 Apr 25 22:32 UTC | HN request time: 0.806s | source

Show context

throwaway150 ◴[12 Apr 25 23:06 UTC] No.43668638[source]▶

Looks cool. But please help me understand. What's to stop AI companies from solving the challenge, completing the proof of work and scrape websites anyway?

replies(6): >>43668690 #>>43668774 #>>43668823 #>>43668857 #>>43669150 #>>43670014 #

marginalia_nu ◴[12 Apr 25 23:41 UTC] No.43668823[source]▶

>>43668638 #

The problem with scrapers in general is the asymmetry of compute resources involved in generating versus requesting a website. You can likely make millions of HTTP requests with the compute required in generating the average response.

If you make it more expensive to request a documents at scale, you make this type of crawling prohibitively expensive. On a small scale it really doesn't matter, but if you're casting an extremely wide net and re-fetching the same documents hundreds of times, yeah it really does matter. Even if you have a big VC budget.

replies(2): >>43669262 #>>43669530 #

charcircuit ◴[13 Apr 25 01:24 UTC] No.43669262[source]▶

>>43668823 #

If you make it prohibitively expensive almost no regular user will want to wait for it.

replies(2): >>43669428 #>>43669586 #

1. xboxnolifes ◴[13 Apr 25 02:34 UTC] No.43669586[source]▶

>>43669262 #

Regular users usually aren't page hopping 10 pages per second. A regular user is usually 100 times less than that.

replies(1): >>43669917 #

2. pabs3 ◴[13 Apr 25 03:38 UTC] No.43669917[source]▶

>>43669586 (TP) #

I tend to get blocked by HN when opening lots of comment pages in tabs with Ctrl+click.

replies(1): >>43670507 #

3. xboxnolifes ◴[13 Apr 25 06:17 UTC] No.43670507[source]▶

>>43669917 #

Yes, HN has a fairly strict slow down policy for commenting. But, that's irrelevant to the context.