←back to thread

646 points blendergeek | 2 comments | | HN request time: 0.427s | source
Show context
grajaganDev ◴[] No.42725460[source]
This keeps generating new pages to keep the crawler occupied.

Looks like this would tarpit any web crawler.

replies(1): >>42725575 #
BryantD ◴[] No.42725575[source]
It would indeed. Note the warning: "There is not currently a way to differentiate between web crawlers that are indexing sites for search purposes, vs crawlers that are training AI models. ANY SITE THIS SOFTWARE IS APPLIED TO WILL LIKELY DISAPPEAR FROM ALL SEARCH RESULTS."
replies(3): >>42725586 #>>42725898 #>>42726004 #
rvnx ◴[] No.42725586[source]
It's actually a great idea to spread malware without leaving traces too, it makes content inspection to be very difficult, view-source: to be broken and most of debugging tools, saving to .har, etc.
replies(1): >>42725842 #
1. bugtodiffer ◴[] No.42725842[source]
how is view source broken
replies(1): >>42726042 #
2. rvnx ◴[] No.42726042[source]
It waits for the whole page to load