(zadzmo.org)

646 points blendergeek | 2 comments | 16 Jan 25 13:57 UTC | HN request time: 0s | source

1. Mr_Bees69 ◴[17 Jan 25 14:55 UTC] No.42738110[source]▶

please add a robots.txt, its quite a d### move to people who build responsible crawlers for fun.

2. marginalia_nu ◴[17 Jan 25 17:48 UTC] No.42741077[source]▶

It's a fairly trivial inconvenience. You can just add something to the effect of the below code, and you'll not get stuck and realistically not skip over crawling anything of value.

  if (response_time > 8 seconds && response_payload < 2048 bytes) {
    extract_links = false;
  }

The odds of a payload that's smaller than the average <head> element taking 20 seconds to load, while containing something worth crawling is fairly low.

↑

Nepenthes is a tarpit to catch AI web crawlers