←back to thread

Blocking LLM crawlers without JavaScript

(www.owl.is)
198 points todsacerdoti | 1 comments | 15 Nov 25 23:30 UTC | HN request time: 0.194s | source
Show context
DeepYogurt ◴[16 Nov 25 03:00 UTC] No.45942404[source]▶
Has anyone done a talk/blog/whatever on how llm crawlers are different than classical crawlers? I'm not up on the difference.
replies(5): >>45942457 #>>45942733 #>>45942771 #>>45942875 #>>45946525 #
1. klodolph ◴[16 Nov 25 03:12 UTC] No.45942457[source]▶
The only real difference that LLM crawlers tend to not respect /robots.txt and some of them hammer sites with some pretty heavy traffic.

The trap in the article has a link. Bots are instructed not to follow the link. The link is normally invisible to humans. A client that visits the link is probably therefore a poorly behaved bot.