←back to thread

Blocking LLM crawlers without JavaScript

(www.owl.is)
198 points todsacerdoti | 1 comments | 15 Nov 25 23:30 UTC | HN request time: 0.471s | source
Show context
SquareWheel ◴[16 Nov 25 01:51 UTC] No.45942060[source]▶
That may work for blocking bad automated crawlers, but an agent acting on behalf of a user wouldn't follow robots.txt. They'd run the risk of hitting the bad URL when trying to understand the page.
replies(2): >>45942461 #>>45942729 #
1. Starlevel004 ◴[16 Nov 25 04:19 UTC] No.45942729[source]▶
Good?