←back to thread

Blocking LLM crawlers without JavaScript

(www.owl.is)
198 points todsacerdoti | 2 comments | 15 Nov 25 23:30 UTC | HN request time: 0.001s | source
Show context
daveoc64 ◴[16 Nov 25 02:08 UTC] No.45942146[source]▶
Seems pretty easy to cause problems for other people with this.

If you follow the link at the end of my comment, you'll be flagged as an LLM.

You could put this in an img tag on a forum or similar and cause mischief.

Don't follow the link below:

https://www.owl.is/stick-och-brinn/

If you do follow that link, you can just clear cookies for the site to be unblocked.

replies(6): >>45942157 #>>45942369 #>>45942605 #>>45943678 #>>45945678 #>>45947272 #
kijin ◴[16 Nov 25 03:49 UTC] No.45942605[source]▶
If a legit user accesses the link through an <img> tag, the browser will send some telling headers. Accept: image/..., Sec-Fetch-Dest: image, etc.

You can also ignore requests with cross-origin referrers. Most LLM crawlers set the Referer header to a URL in the same origin. Any other origin should be treated as an attempted CSRF.

These refinements will probably go a long way toward reducing unintended side effects.

replies(1): >>45943611 #
1. Terr_ ◴[16 Nov 25 08:34 UTC] No.45943611[source]▶
Even if we somehow guard against <img> and <iframe> and <script> etc., someone on a webforum that supports formatting links could just trick viewers into clicking a normal <a>, thinking they're accessing a funny picture or whatever.

A bunch of CSRF/nonce stuff could apply if it were a POST instead...

It may be more-effective to make the link unique and temporary, expiring fast enough that "hey, click this" is limited in its effectiveness. That might reduce true-positive detections of a bot that delays its access though.

replies(1): >>45944658 #
2. kijin ◴[16 Nov 25 12:37 UTC] No.45944658[source]▶
If it were my forum, I would just strip out any links to the honeypot URL. I have full control over who can post links to what URL, after all.

You could use a URL shortener to bypass the ban, but then you'll be caught by the cross-origin referrer check.