←back to thread

Blocking LLM crawlers without JavaScript

(www.owl.is)
198 points todsacerdoti | 7 comments | 15 Nov 25 23:30 UTC | HN request time: 0.422s | source | bottom
1. bakigul ◴[16 Nov 25 10:59 UTC] No.45944142[source]▶
Why do people want to block LLM crawlers? Everyone wants to be visible in GPT, Claude, and others, don’t they?
replies(6): >>45944209 #>>45944222 #>>45944288 #>>45944466 #>>45946287 #>>45947378 #
2. aiven ◴[16 Nov 25 11:13 UTC] No.45944209[source]▶
- If LLM knows about your content, people don't really need to visit your site - LLM crawlers can be pretty aggressive and eat up a lot of traffic - Google will not suggest its misleading "summaries" from its search for your site - Some people just hate LLMs that much ¯\_(ツ)_/¯
3. saltysalt ◴[16 Nov 25 11:18 UTC] No.45944222[source]▶
Exactly! With many people moving to AI-driven search, if you are not in their indexes you traffic will reduce even further.

And I say this as someone who built a search engine with no AI: I know my audience for that service is very niche, with the vast majority of people using AI search because it's more convenient.

4. suckler ◴[16 Nov 25 11:31 UTC] No.45944288[source]▶
You're a funny guy.
5. ThomasMidgley ◴[16 Nov 25 12:03 UTC] No.45944466[source]▶
No
6. bo1024 ◴[16 Nov 25 16:27 UTC] No.45946287[source]▶
One issue is just the sheer amount of such traffic.
7. xgulfie ◴[16 Nov 25 18:47 UTC] No.45947378[source]▶
If someone doesn't want to visit my website then why would I provide my website to them