Most active commenters

    ←back to thread

    770 points ta988 | 18 comments | | HN request time: 0.001s | source | bottom
    Show context
    markerz ◴[] No.42551173[source]
    One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-...

    It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again.

    My solution was to add a Cloudflare rule to block requests from their User-Agent. I also added more nofollow rules to links and a robots.txt but those are just suggestions and some bots seem to ignore them.

    Cloudflare also has a feature to block known AI bots and even suspected AI bots: https://blog.cloudflare.com/declaring-your-aindependence-blo... As much as I dislike Cloudflare centralization, this was a super convenient feature.

    replies(14): >>42551260 #>>42551410 #>>42551412 #>>42551513 #>>42551649 #>>42551742 #>>42552017 #>>42552046 #>>42552437 #>>42552763 #>>42555123 #>>42562686 #>>42565119 #>>42572754 #
    MetaWhirledPeas ◴[] No.42551742[source]
    > Cloudflare also has a feature to block known AI bots and even suspected AI bots

    In addition to other crushing internet risks, add wrongly blacklisted as a bot to the list.

    replies(4): >>42551773 #>>42552921 #>>42562510 #>>42564887 #
    kmeisthax ◴[] No.42562510[source]
    This is already a thing for basically all of the second[0] and third worlds. A non-trivial amount of Cloudflare's security value is plausible algorithmic discrimination and collective punishment as a service.

    [0] Previously Soviet-aligned countries; i.e. Russia and eastern Europe.

    replies(5): >>42562599 #>>42563762 #>>42564357 #>>42566973 #>>42567500 #
    1. ls612 ◴[] No.42562599[source]
    People hate collective punishment because it works so well.
    replies(5): >>42562792 #>>42563310 #>>42563642 #>>42563761 #>>42563805 #
    2. eckesicle ◴[] No.42562792[source]
    Anecdatally, by default, we now block all Chinese and Russian IPs across our servers.

    After doing so, all of our logs, like ssh auth etc, are almost completely free and empty of malicious traffic. It’s actually shocking how well a blanket ban worked for us.

    replies(5): >>42562837 #>>42563023 #>>42567554 #>>42569757 #>>42574189 #
    3. macintux ◴[] No.42562837[source]
    ~20 years ago I worked for a small IT/hosting firm, and the vast majority of our hostile traffic came from APNIC addresses. I seriously considered blocking all of it, but I don’t think I ever pulled the trigger.
    4. panic ◴[] No.42563310[source]
    Works how? Are these blocks leading to progress toward solving any of the underlying issues?
    replies(2): >>42563743 #>>42573501 #
    5. ◴[] No.42563642[source]
    6. forgetfreeman ◴[] No.42563743[source]
    It's unclear that there are actors below the regional-conglomerate-of-nation-states level that could credibly resolve the underlying issues, and given legislation and enforcement regimes sterling track record of resolving technological problems realistically it seems questionable that solutions could exist in practice. Anyway this kind of stuff is well outside the bounds of what a single org hosting an online forum could credibly address. Pragmatism uber alles.
    7. anonym29 ◴[] No.42563761[source]
    Innocent people hate being punished for the behavior of other people, whom the innocent people have no control over.*

    FTFY.

    replies(1): >>42563952 #
    8. saagarjha ◴[] No.42563805[source]
    Putting everyone in jail also works well to prevent crime.
    replies(1): >>42575772 #
    9. zdragnar ◴[] No.42563952[source]
    The phrase "this is why we can't have nice things" springs to mind. Other people are the number one cause of most people's problems.
    replies(1): >>42564188 #
    10. thwarted ◴[] No.42564188{3}[source]
    Tragedy of the Commons Ruins Everything Around Me.
    11. TacticalCoder ◴[] No.42567554[source]
    > Anecdatally, by default, we now block all Chinese and Russian IPs across our servers.

    This. Just get several countries' entire IP address space and block these. I've posted I was doing just that only to be told that this wasn't in the "spirit" of the Internet or whatever similar nonsense.

    In addition to that only allow SSH in from the few countries / ISPs legit trafic shall legitimately be coming from. This quiets the logs, saves bandwidth, saves resources, saves the planet.

    replies(1): >>42570683 #
    12. citrin_ru ◴[] No.42569757[source]
    Being slightly annoyed by noise in SSH logs I’ve blocked APNIC IPs and now see a comparable number of brute force attempts from ARIN IPs (mostly US ones). Geo blocks are totally ineffective against TAs which use a global network of proxies.
    13. brianwawok ◴[] No.42570562{3}[source]
    That is not at all the reason for the great firewall.
    14. xp84 ◴[] No.42570683{3}[source]
    I agree with your approach. It’s easy to empathize with innocent people in say, Russia, blocked from a site which has useful information to them. However the thing these “spirit/openness” people miss is that many sites have a narrow purpose which makes no sense to open it up to people across the world. For instance, local government. Nobody in India or Russia needs to see the minutes from some US city council meeting, or get building permit information. Likewise with e-commerce. If I sell chocolate bars and ship to US and Canada, why wouldn’t I turn off all access from overseas? You might say “oh, but what if some friend in $COUNTRY wants to order a treat for someone here?” And the response to that is always “the hypothetical loss from that is minuscule compared to the cost of serving tons of bot traffic as well as possible exploits those bots might do.

    (Yes, yes, VPNs and proxies exist and can be used by both good and bad actors to evade this strategy, and those are another set of IPs widely banned for the same reason. It’s a cat and mouse game but you can’t argue with the results)

    15. victorbjorklund ◴[] No.42573501[source]
    The underlying issue is that countries like russia support abuse like this. So by blocking them perhaps the people there will demand that their govt stops supporting crimes and absuse so that they can be allowed back into the internet.

    (In the case of russians though i guess they will never change)

    replies(1): >>42574222 #
    16. ◴[] No.42574189[source]
    17. petre ◴[] No.42574222{3}[source]
    > people there will demand that their govt stops supporting crimes and absuse so that they can be allowed back into the internet

    Sure. It doesn't work that way, not in Russia or China. First they have to revert back to 1999 when Putin took over. Then they have to extradite criminals and crack down on cybercrime. Then maybe they could be allowed back onto the open Internet.

    In my country one would be exradited to the US in no time. In fact the USSS came over for a guy who had been laundering money through BTC from a nearby office. Not a month passed and he got extradited to the US, never to be heard from again.

    18. singleshot_ ◴[] No.42575772[source]
    Having a door with a lock on it prevents other people from committing crime in my house. This metaphor has the added benefit of making some amount of sense in context.