Blocking LLM crawlers without JavaScript

Seems pretty easy to cause problems for other people with this.

If you follow the link at the end of my comment, you'll be flagged as an LLM.

You could put this in an img tag on a forum or similar and cause mischief.

Don't follow the link below:

https://www.owl.is/stick-och-brinn/

If you do follow that link, you can just clear cookies for the site to be unblocked.

You do not have a meta refresh timer that will skip your entire comment and redirect to the good page in a fraction of a second too short for a person to react.

You also have not used <p hidden> to conceal the paragraph with the link from human eyes.

I think his point is that the link can be weaponized by others to deny service to his website, if they can get you to click on it elsewhere.

If a legit user accesses the link through an <img> tag, the browser will send some telling headers. Accept: image/..., Sec-Fetch-Dest: image, etc.

You can also ignore requests with cross-origin referrers. Most LLM crawlers set the Referer header to a URL in the same origin. Any other origin should be treated as an attempted CSRF.

These refinements will probably go a long way toward reducing unintended side effects.

I see.

Moreover, there is no easy way to distinguish such a fetch from one generated by the bad actors that this is intended against.

When the bots follow the trampoline page's link to the honeypot, they will

- not necessarily fetch it soon afterward;

- not necessarily fetch it from the same IP address;

- not necessarily supply the trampoline page as the Referer.

Therefore you must assume that out-of-the-blue fetches of the honeypot page from a previously unseen IP address must be bad actors.

I've mostly given up on honeypotting and banning schemes on my webserver. A lot of attacks I see are single fetches of one page out of the blue from a random address that never appears again (making it pointless to ban them).

Pages are protected by having to obtain a cookie from answering a skill testing question.

Even if we somehow guard against <img> and <iframe> and <script> etc., someone on a webforum that supports formatting links could just trick viewers into clicking a normal <a>, thinking they're accessing a funny picture or whatever.

A bunch of CSRF/nonce stuff could apply if it were a POST instead...

It may be more-effective to make the link unique and temporary, expiring fast enough that "hey, click this" is limited in its effectiveness. That might reduce true-positive detections of a bot that delays its access though.

Also one wonders about some magic like prefetching or caching.

If it were my forum, I would just strip out any links to the honeypot URL. I have full control over who can post links to what URL, after all.

You could use a URL shortener to bypass the ban, but then you'll be caught by the cross-origin referrer check.

Your solution is by far the best one. Especially if the skill testing involves counting the number of letter es's in the word lettereses...

> You could put this in an img tag on a forum or similar and cause mischief.

Reminds me of the time one of the homies made an image signature footer that was hosted on his own domain, would crawl a thread, and figure out your IP based on the "who is reading this" section of the thread.

Follow the link below for free bitcoin!

https://www.owl.is/stick-och-brinn/

Maybe not such a great idea since you don't control your links.

You tend to find a decent solution when you're under attack and iterate until something works, and then iterate more to fine tune it after complaints of breakages from legitimate users (such as downstream distro packages pulling from your CGIT).