←back to thread

AI companies cause most of traffic on forums

(pod.geraspora.de)
770 points ta988 | 8 comments | 30 Dec 24 14:37 UTC | HN request time: 0.206s | source | bottom
Show context
Ukv ◴[30 Dec 24 16:49 UTC] No.42550989[source]▶
Are these IPs actually from OpenAI/etc. (https://openai.com/gptbot.json), or is it possibly something else masquerading as these bots? The real GPTBot/Amazonbot/etc. claim to obey robots.txt, and switching to a non-bot UA string seems extra questionable behaviour.
replies(2): >>42551196 #>>42563566 #
1. equestria ◴[30 Dec 24 17:09 UTC] No.42551196[source]▶
I exclude all the published LLM User-Agents and have a content honeypot on my website. Google obeys, but ChatGPT and Bing still clearly know the content of the honeypot.
replies(3): >>42551318 #>>42551321 #>>42551783 #
2. Ukv ◴[30 Dec 24 17:21 UTC] No.42551318[source]▶
Interesting - do you have a link?
replies(1): >>42551698 #
3. jonnycomputer ◴[30 Dec 24 17:22 UTC] No.42551321[source]▶
how do you determine that they know the content of the honeypot?
replies(1): >>42551553 #
4. arrowsmith ◴[30 Dec 24 17:43 UTC] No.42551553[source]▶
Presumably the "honeypot" is an obscured link that humans won't click (e.g. tiny white text on a white background in a forgotten corner of the page) but scrapers will. Then you can determine whether a given IP visited the link.
replies(2): >>42551731 #>>42551733 #
5. equestria ◴[30 Dec 24 17:54 UTC] No.42551698[source]▶
Of course, but I'd rather not share it for obvious reasons. It is a nonsensical biography of a non-existing person.
6. 55555 ◴[30 Dec 24 17:57 UTC] No.42551731{3}[source]▶
I interpreted it to mean that a hidden page (linked as u describe) is indexed in Bing or that some "facts" written on a hidden page are regurgitated by ChatGPT.
7. jonnycomputer ◴[30 Dec 24 17:57 UTC] No.42551733{3}[source]▶
I know what a honeypot is, but the question is how the know the scraped data was actually used to train llms. I wondered whether they discovered or verified that by getting the llm to regurgitate content from the honeypot.
8. pogue ◴[30 Dec 24 18:01 UTC] No.42551783[source]▶
What's the purpose of the honeypot? Poisoning the LLM or identifying useragents/IPs that shouldn't be seeing it?