←back to thread

AI scrapers request commented scripts

(cryptography.dog)
257 points ColinWright | 3 comments | 31 Oct 25 15:44 UTC | HN request time: 0s | source
Show context
Noumenon72 ◴[31 Oct 25 17:25 UTC] No.45774469[source]▶
It doesn't seem that abusive. I don't comment things out thinking "this will keep robots from reading this".
replies(2): >>45774493 #>>45774628 #
mostlysimilar ◴[31 Oct 25 17:42 UTC] No.45774628[source]▶
The article mentions using this as a means of detecting bots, not as a complaint that it's abusive.

EDIT: I was chastised, here's the original text of my comment: Did you read the article or just the title? They aren't claiming it's abusive. They're saying it's a viable signal to detect and ban bots.

replies(3): >>45774645 #>>45774743 #>>45776844 #
1. woodrowbarlow ◴[31 Oct 25 17:51 UTC] No.45774743[source]▶
the first few words of the article are:

> Last Sunday I discovered some abusive bot behaviour [...]

replies(2): >>45774770 #>>45774783 #
2. mostlysimilar ◴[31 Oct 25 17:53 UTC] No.45774770[source]▶
> The robots.txt for the site in question forbids all crawlers, so they were either failing to check the policies expressed in that file, or ignoring them if they had.
3. foobarbecue ◴[31 Oct 25 17:54 UTC] No.45774783[source]▶
Yeah but the abusive behavior is ignoring robots.txt and scraping to train AI. Following commented URLs was not the crime, just evidence inadvertently left behind.