←back to thread

255 points ColinWright | 1 comments | | HN request time: 0.201s | source
Show context
Noumenon72 ◴[] No.45774469[source]
It doesn't seem that abusive. I don't comment things out thinking "this will keep robots from reading this".
replies(2): >>45774493 #>>45774628 #
michael1999 ◴[] No.45774493[source]
Crawlers ignoring robots.txt is abusive. That they then start scanning all docs for commented urls just adds to the pile of scummy behaviour.
replies(1): >>45774679 #
1. tveyben ◴[] No.45774679[source]
Human behavior is interesting - me, me, me…