←back to thread

Blocking LLM crawlers without JavaScript

(www.owl.is)
198 points todsacerdoti | 3 comments | 15 Nov 25 23:30 UTC | HN request time: 0.459s | source
1. petesergeant ◴[16 Nov 25 02:12 UTC] No.45942169[source]▶
I wish blockers would distinguish between crawlers that index, and agentic crawlers serving an active user's request. npm blocking Claude Code is irritating
replies(2): >>45942313 #>>45942467 #
2. specialp ◴[16 Nov 25 02:41 UTC] No.45942313[source]▶
Agentic crawlers are worse. I run a primary source site and the ai "thinking" user agents will hit your site 1000+ times in a minute at any time of the day
3. klodolph ◴[16 Nov 25 03:15 UTC] No.45942467[source]▶
I think of those two, agentic crawlers are worse.