←back to thread

Faking a JPEG

(www.ty-penguin.org.uk)
388 points todsacerdoti | 1 comments | 11 Jul 25 22:57 UTC | HN request time: 0.213s | source
Show context
tomsmeding ◴[12 Jul 25 07:47 UTC] No.44540147[source]▶
They do have a robots.txt [1] that disallows robot access to the spigot tree (as expected), but removing the /spigot/ part from the URL seems to still lead to Spigot. [2] The /~auj namespace is not disallowed in robots.txt, so even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice.

[1]: https://www.ty-penguin.org.uk/robots.txt

[2]: https://www.ty-penguin.org.uk concatenated with /~auj/cheese (don't want to create links there)

replies(2): >>44540567 #>>44540604 #
josephg ◴[12 Jul 25 09:16 UTC] No.44540567[source]▶
> even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice.

So? What duty do web site operators have to be "nice" to people scraping your website?

replies(2): >>44540606 #>>44540718 #
suspended_state ◴[12 Jul 25 09:46 UTC] No.44540718[source]▶
The point is that not every web crawler is out there to scrape websites.
replies(1): >>44541486 #
andybak ◴[12 Jul 25 12:06 UTC] No.44541486[source]▶
Unless you define "scrape" to be inherently nefarious - then surely they are? Isn't the definition of a web crawler based on scraping websites?
replies(1): >>44543799 #
1. suspended_state ◴[12 Jul 25 18:03 UTC] No.44543799[source]▶
I think that web scraping is usually understood as the act of extracting information of a website for ulterior self-centered motives. However, it is clear that this ulterior motive cannot be assessed by a website owner. Only the observable behaviour of a data collecting process can be categorized as morally good or bad. While the bad behaving people are usually also the ones with morally wrong motives, one doesn't entail the other. I chose to qualify the bad behaving ones as scrapers, and the good behaving ones as crawlers.

That being said, the author is perhaps concerned by the growing amount of collecting process, which carries a toll on his server, and thus chose to simply penalize them all.