←back to thread

597 points classichasclass | 1 comments | | HN request time: 0.208s | source
1. ricardo81 ◴[] No.45012469[source]
Any respectable web scale crawler(/scraper) should have reverse DNS so that it can automatically be blocked.

Though it would seem all bets are off and anyone will scrape anything. Now we're left with middlemen like cloudflare that cost people millions of hours of time ticking boxes to prove they're human beings.