I'm always a little surprised to see how many people take robots.txt seriously on HN. It's nice to see so many folks with good intentions.
However, it's obviously not a real solution. It depends on people knowing about it, and adding the complexity of checking it to their crawler. Are there other more serious solutions? It seems like we've heard about "micropayments" and "a big merkle tree of real people" type solutions forever and they've never materialized.
replies(2):