←back to thread

Blocking LLM crawlers without JavaScript

(www.owl.is)
198 points todsacerdoti | 1 comments | 15 Nov 25 23:30 UTC | HN request time: 0.317s | source
Show context
snehesht ◴[17 Nov 25 01:09 UTC] No.45950013[source]▶
Isn’t this easy for LLMs to avoid by passing an instruction to ignore any hidden links ?
replies(1): >>45950119 #
krackers ◴[17 Nov 25 01:34 UTC] No.45950119[source]▶
Companies mass crawling don't use LLMs for crawling itself, that would be too expensive.
replies(1): >>45976363 #
1. snehesht ◴[19 Nov 25 06:05 UTC] No.45976363[source]▶
Make sense, but doesn't necessarily have to be an llm, just a regular dom parser will be able to tell whether an element is visible or hidden.