←back to thread

646 points blendergeek | 1 comments | | HN request time: 0s | source
Show context
taikahessu ◴[] No.42726337[source]
We had our non-profit website drained out of bandwidth and site closed temporarily (!!) from our hosting deal because of Amazon bot aggressively crawling like ?page=21454 ... etc.

Gladly Siteground restored our site without any repercussions as it was not our fault. Added Amazon bot into robots.txt after that one.

Don't like how things are right now. Is a tarpit the solution? Or better laws? Would they stop the chinese bots? Should they even? I don't know.

replies(4): >>42726365 #>>42735381 #>>42740706 #>>42743952 #
bee_rider ◴[] No.42740706[source]
It is too bad we don’t have a convention already for the internet:

User/crawler: I’d like site

Server: ok that’ll be $.02 for me to generate it and you’ll have to pay $.01 in bandwidth costs, plus whatever your provider charges you

User: What? Obviously as a human I don’t consume websites so fast that $.03 will matter to me, sure, add it to my cable bill.

Crawler: Oh no, I’m out of money, (business model collapse).

replies(1): >>42741245 #
1. jmholla ◴[] No.42741245[source]
I think that's a terrible idea, especially with ISP monopolies that love gouging their customers. They have a demonstrable history of markups well beyond their means.

And I hope you're pricing this highly. I don't know about you, but I would absolutely notice $.03 a site on my bill, just from my human browsing.

In fact, I feel like this strategy would further put the Internet in the hands of the aggregators as that's the one site you know you can get information from, so long term that cost becomes a rounding error for them as people are funneled to their AI as their memberships are cheaper than accessing the rest of the web.