(chatgpt.com)

763 points easton | 1 comments | 21 Oct 25 17:18 UTC | HN request time: 0s | source

Show context

scoofy ◴[21 Oct 25 17:41 UTC] No.45658890[source]▶

>>45658479 (OP) #

This is a fun way to get around robots.txt "Disallow: /"

replies(2): >>45659438 #>>45659836 #

SeanAnderson ◴[21 Oct 25 18:13 UTC] No.45659438[source]▶

>>45658890 #

There is no law that says you have to respect robots.txt. It's just a suggestion.

replies(1): >>45659628 #

1. scoofy ◴[21 Oct 25 18:26 UTC] No.45659628[source]▶

>>45659438 #

For the websites that ChatGPT wants to scrape -- Reddit immediately comes to mind -- it's not an issue of law, it's an issue of "the infrastructure now exists to prevent you from doing that."

↑

ChatGPT Atlas