Cloudflare's Browser Intergrity Check/Verification/Challenge feature used by many websites, is denying access to users of non-mainstream browsers like Pale Moon.
Users reports began on January 31:
https://forum.palemoon.org/viewtopic.php?f=3&t=32045
This situation occurs at least once a year, and there is no easy way to contact Cloudflare. Their "Submit feedback" tool yields no results. A Cloudflare Community topic was flagged as "spam" by members of that community and was promptly locked with no real solution, and no official response from Cloudflare:
https://community.cloudflare.com/t/access-denied-to-pale-moo...
Partial list of other browsers that are being denied access:
Falkon, SeaMonkey, IceCat, Basilisk.
Hacker News 2022 post about the same issue, which brought attention and had Cloudflare quickly patching the issue:
https://news.ycombinator.com/item?id=31317886
A Cloudflare product manager declared back then: "...we do not want to be in the business of saying one browser is more legitimate than another."
As of now, there is no official response from Cloudflare. Internet access is still denied by their tool.
On one hand, I get the annoying "Verify" box every time I use ChatGPT (and now due its popularity, DeepSeek as well).
On the other hand, without Cloudflare I'd be seeing thousands of junk requests and hacking attempts everyday, people attempting credit card fraud, etc.
I honestly don't know what the solution is.
Robots went out of control, whether malicious or the AI scrapers or the Clearview surveillance kind; users learned to not trust random websites; SEO spam ruined search, the only thing that made a decentralized internet navigable; nation state attacks became a common occurrence; people prefer a few websites that do everything (Facebook becoming an eBay competitor). Even if it were possible to set rules banning Clearview or AI training, no nation outside of your own will follow them; an issue which even becomes a national security problem (are you sure, Taiwan, that China hasn't profiled everyone on your social media platforms by now?)
There is no solution. The dream itself was not sustainable. The only solution is either a global moratorium of understanding which everyone respectfully follows (wishful thinking, never happening); or splinternetting into national internets with different rules and strong firewalls (which is a deal with the devil, and still admitting the vision failed).
Yup!
> I honestly don't know what the solution is.
Force law enforcement to enforce the laws.
Or else, block the countries that don't combat fraud. That means... China? Hey isn't there a "trade war" being "started"? It sure would be fortunate if China (and certain other fraud-friendly countries around Asia/Pacific) were blocked from the rest of the Internet until/unless they provide enforcement and/or compensation their fraudulent use of technology.
The thing is that these tools are generally used to further entrench power that monopolies, duopolies, and cartels already have. Example: I've built an app that compares grocery prices as you make a shopping list, and you would not believe the extent that grocers go to to make price comparison difficult. This thing doesn't make thousands or even hundreds of requests - maybe a few dozen over the course of a day. What I thought would be a quick little project has turned out to be wildly adversarial. But now spite driven development is a factor so I will press on.
It will always be a cat and mouse game, but we're at a point where the cat has a 46 billion dollar market cap and handles a huge portion of traffic on the internet.
Though annoying, it's tolerable. It seemed like a fair solution. Blocking doesn't.
This usually includes people making a near-realtime updated perfect copy of your site and serving that copy for either scam or middle-manning transactions or straight fraud.
Having a clear category of "good bots" from either a verified or accepted companies would help for these cases. Cloudflare has such a system I think, but then a new search engine would have to go to each and every platform provider to make deals and that also sounds impossible.
They ignored robots.txt (claimed not to, but I blacklisted them there and they didn't stop) and started randomly generating image paths. At some point /img/123.png became /img/123.png?a=123 or whatever, and they just kept adding parameters and subpaths for no good reason. Nginx dutifully ignored the extra parameters and kept sending the same images files over and over again, wasting everyone's time and bandwidth.
I was able to block these bots by just blocking the entire IP range at the firewall level (for Huawei I had to block all of China Telecom and later a huge range owned by Tencent for similar reasons).
I have lost all faith in scrapers. I've written my own scrapers too, but almost all of the scrapers I've come across are nefarious. Some scour the internet searching for personal data to sell, some look for websites to send hack attempts at to brute force bug bounty programs, others are just scraping for more AI content. Until the scraping industry starts behaving, I can't feel bad for people blocking these things even if they hurt small search engines.
To make matters worse, I suspect that not even a splinternet can save it. It needs a new foundation, preferably one that wasn't largely designed before security was a thing.
Federation is probably a good start, but it should be federated well below the application layer.
The solution is good security-- Cloudflare only cuts down on the noise. I'm looking at junk requests and hacking attempts flow through to my sites as we speak.
Countries, whether it be Ukraine or Taiwan, can't risk other countries harvesting their social media platforms for the mother of all purges. I never assume that anything that happened historically can never happen again - no Polish Jew would have survived the Nazis with this kind of information theft. Add AI into the mix, and wiping out any population is as easy as baking pie.
Countries are tired of bad behavior. Just ask my grandmother, who has had her designs stolen and mass produced from China. Not just companies - many free and open source companies cannot survive with such reckless competition. Can Prusa survive a world where China takes, but never gives? How many grandmothers does it take being scammed? How many educational systems containing data on minors need to be stolen? The MPAA and RIAA has been whining for years about the copyright problem, and while we laugh at them, never underestimate them. The list goes on and on.
Startups are tired of paying Cloudflare or AWS protection money, and trying to evade the endless sea of SEO spam. How can a startup compete with Google with so much trash and no recourse? Who can build a new web browser, and be widely accepted as being a friendly visitor? Who can build a new social media platform, without the experience and scale to know who is friend or foe?
Now we have AI, gasoline and soon to be dynamite on the fire. For the first time ever, a malicious country can VPN into the internet of a friendly nation, track down all critics on their social media, and destroy their lives in a real world attack (physical or virtual). We are only beginning to see this in Ukraine - are we delusional enough to believe that the world is past warfare? For the first time, anyone in the world could make nudes of women and share them online, from a location where they'll probably never be taken down. If a Russian company offered nudes as a service to American customers with cryptocurrency payments and a slick website that went viral, do you think tolerance is a winning political position?
I think a decent idea is, we need to bring personal accountability back into the equation. That's how an open-trust network works, and we know that, because that's how society works. You don't "trust" that someone walking by your car won't take a shit in your open window: they could. But there are consequences for that. We need rock solid data security policies that apply to anyone who does business, hosts content, handles user data online, and people need to use their actual names, actual addresses, actual phone numbers, etc. etc. in order to interact with it. I get that there are many boons to be had with the anonymity the Internet offers, but it also enables all of the horseshit we all hate. A spammer can spam explicitly because their ISP doesn't care that they do, email servers don't have their actual information, and in the odd event they are caught and are penalized, it's fucking trivial to circumvent it. Buy a new AWS instance, run a script to setup your spam box, upload your database of potential victims, and boom, you're off.
A lot of tech is already drifting this way. What is HTTPS at it's core if not a way to verify you are visiting the real Chase.com? How many social networking sites now demand all kinds of information, up to and including a photo of your driver's license? Why are we basically forbidden now by good practice from opening links in texts and emails? Because too many people online are anonymous, can't be trusted, and are acting maliciously. Imagine how much BETTER the Internet would be if when you fucked around, you could be banned entirely? No more ban evasion, ever.
I get that this is a controversial opinion, but fundamentally, I don't think the Internet can function for much longer while being this free. It's too free, and we have too many opportunistic assholes in it for it to remain so.
Fundamentally it's adversarial, so expecting a single simple concept to properly cover even half of the problematic requests is unrealistic.
this is wrong.
if someone can use your site they can use stolen cards, and bots doing this will not be stopped by them.
cloudflare only raises the cost of doing it, it may make scrapping a million of product pages unprofitable but that doesn't apply to cc fraud yet.
I used to work at a company that did auto inspections. (e.x. if you turned a lease in, did a trade in on a used car, private party, etc.)
Because of that, we had a server that contained 'condition reports', as well as the images that went through those condition reports.
Mind you, sometimes condition reports had to be revised. Maybe a photo was bad, maybe the photos were in the wrong order, etc.
It was a perfect storm:
- The Image caching was all inmem
- If an image didn't exist, the server would error with a 500
- IIS was set up such that too many errors caused a recycle
- Some scraper was working off a dataset (that ironically was 'corrected' in an hour or so) but contained an image that did not exist.
- The scraper, instead of eventually 'moving on' would keep retrying the URL.
It was the only time that org had an 'anyone who thinks they can help solve please attend' meeting at the IT level.
> and you would not believe the extent that grocers go to to make price comparison difficult. This thing doesn't make thousands or even hundreds of requests - maybe a few dozen over the course of a day.
Very true. I'm reminded of Oren Eini's tale of building an app to compare grocery prices in Israel, where apparently mandated supermarket chains to publish prices [0]. On top of even the government mandate for data sharing appearing to hit the wrong over/under for formatting, There's the constant issue of 'incomparabilities'.
And it's weird, because it immediately triggered memories of how 20-ish years ago, one of the most accessible Best Buy's was across the street from a Circuit City, but good luck price matching because the stores all happened to sell barely different laptops/desktops (e.x. up the storage but use a lower grade CPU) so that nobody really had to price match.
[0] - https://ayende.com/blog/170978/the-business-process-of-compa...
So instead of banning America, report the IP addresses to their American hosts for spam and malicious intent. If the host refuses to do anything, report it to law enforcement. If law enforcement doesn't do anything... then you're proving my point.
It's gonna get even worse. Walmart & Kroger are implementing digital price tags, so whatever you see on the website will probably (purposefully?) be out of date by the time you get to the store.
Stores don't want you to compare.
And yea, I imagine dynamic pricing will make things even more complicated.
That being said, that's why this feature isn't built into the billion shopping list apps that are out there. Because it's a pain.
It stops "card testing" where someone has bought or stolen a large number of cards and need verify which are still good. The usual technique is to cycle through all the cards on a smaller site selling something cheap (a $3 ebook for example). The problem is that the high volume of fraud in a short time span will often get the merchant account or payment gateway account shut down, cutting off legitimate sales.
As a consumer, you should also be suspicious of a mysterious low value charge on your card because it could be the prelude to much larger charges.
I'm not sure this is a good is a good example. I believe a majority of Polish Jewish survivors were those who fled into parts of soviet union not occupied by nazis(some were sent to gulags but this was still much better chance to survive then those who stayed in Poland). Another large portion were in concentration camps and hadn't been killed yet. And I believe 60,000 or less are estimated to have hid in Poland through the war. It's unlikely many remained in their pre war identities and simply concealed their Jewishness and managed to survive.
well, for starters, if you're using cloudflare to block otherwise benign traffic, just because you're worried about some made... up....
> On the other hand, without Cloudflare I'd be seeing thousands of junk requests and hacking attempts everyday, people attempting credit card fraud, etc.
well damn, if you're using it because otherwise you'd be exposing your users to active credit card fraud... I guess the original suggestion to only ban traffic once you find it to be abusive, and then only by subnet, doesn't really apply for you.
I wanna suggest using this as an excuse to learn how not to be a twat (the direction cf is moving towards more and more), where for most sites 20% of the work will get you 80% of the results... but dealing with cc fraud, you're adversaries are already on the more advanced side, and that becomes a lot harder to prevent... rather than catch and stop after the fact.
Balancing the pervasive fear mongering with sensible rules is hard. Not because it's actually hard, but because that's the point of the FUD. To create the perception of a problem where there isn't one. With a few exceptions, a WAF doesn't provide meaningful benefits. It only serves to lower the number of log entries, it rarely ever reduces the actual risk.
Anyway, no, this guessing game isn't the solution to stolen bank details, the solution is for the payment provider to authenticate the account holder beyond merely entering a public number, especially if they suddenly see a flood of transactions from this one merchant as you describe. They can decide to ask for a second factor: send the person an SMS/email, ask to generate an authenticator code, whatever it is they've got on file beyond your card/account number. Anything else is just guesswork
I host many webpages and this is exactly it. Anyone is welcome to use the websites I host. There is no CDN, your TLS session terminates at the endpoint (end to end encryption). May be a bit slower for the pages having static assets if you're coming from outside of Europe, but the pages are light anyway (no 2 MB JavaScript blobs)
It works so well and is very secure. You get to the checkout page on a website, click a link. If you’re on your phone, it hotlinks to open your banking app. If you’re on desktop, it shows a QR code which does the same.
When your bank app opens, it says “would you like to make this €28 payment to Business X?” And you click either yes or no on the app. You never even need to enter a card in the website!
You can also send money to other people instantly the same way, so it’s perfect for something like buying a used item from someone else.
Plus the whole IBAN system which makes it all possible!
Specifically, I use fail2ban to count the 404s and ban the IP temporarily when certain threshold is exceeded in a given time frame. Every time I check fail2ban stats it has hundreds of IPs blocked.
You block all VPN users then, and currently many countries have some kind of censorship, please don't do that. I use a personal VPN for over 5 years and that's annoying.
I understand the other side and captcha/POW captchas/additional checks is okay. But give people a choice to be private/non-censorable.
Enabling/disabling a VPN each minute to access the non-censored local site which blocks datacenters IPs, then bringing it back again for the general surfing is a bit of a hell.
BTW, how they should report it, if they are a small business/physical person without lawyers? Does US police have some kind of online hotline to report US criminals for foreigners or smth?
What I mean is that it's better to give VPN users the choice to solve captchas instead of being banned completely.
In other words, knowing who someone is isn't strictly necessary, provided they have "skin the game" to encourage proper behavior.
When the government really cares, it can put all its resources to solve any particular problem. Though obviously that comes at the cost of reassigning resources from other tasks. Sadly it's impossible to assign all resources to solve every problem all at once.
I get that there might be some feeling of righteous justice that comes from removing these entries from your Nginx logs, but it also seems like there's a lot of self-induced stress that comes from monitoring failed Nginx and ssh logs.
Maybe 10% of the time I make a purchase online, it shows me a screen where it says it's waiting for my bank to verify, I'll have to input a code or accept a notification or something.
A solid half the time it fails. Either the site decides the transaction was rejected before I even get a chance to respond (within seconds), or I just don't get any notification or code or anything, or I do authorize it and it still gets rejected.
Cloudflare cuts down on the noise, but also helps does the work of preventing scrapers, people who re-sell your site wholesale, and cutting down on the noise also means cutting down on the cost of network requests.
It also can help where security is lax. You should have measures against credential stuffing, but if you don't, Cloudflare might prevent (some) of your users from being hacked. Which isn't good enough, but is better than no mitigation at all.
I don't use Cloudflare personally, but I won't dismiss it wholesale. I understand why people use it.
That sounds like the solution, that sounds like good security.
I guess you get some security since each party that you transfer to must have their identity verified with a bank, so you could always get the police involved fairly easily
The iDeal website page on security [2] is in Dutch, but it translates to roughly:
> Before you make a purchase, make sure that the webshop or business is a reliable party. For example, you can read experiences of other consumers about webshops on comparison sites. Or you can use a Google search to check what is said (in reviews) about a webshop on the internet. Also check the overview of the police with known rogue trading parties and the page check seller data. Before making a purchase, always use the following rule of thumb: if something is too good to be true, don't do it.
But in the US there are so many credit card providers, each one seems to do it differently, and the UX flows just break. And it seems difficult for a site to even test, and how will you even figure out if it's the provider or network or merchant or notification that's failing?
Bots are a fact of life. Secure your site properly, follow good practices, set up notifications for important things, log stuff, but don't look at the logs unless you have a reason to look at the logs.
Having run web servers forever, this is simply normal. What's not normal is blindly trusting a megacorporation to make my logs quiet. What're they doing? Who are they blocking? What guidelines do they use? Nobody, except them, knows.
It's why I self-host email. Sure, you might feel safe because most people use Gmail or Outlook, and therefore if there are problems, you can point the finger at them, but what if you want to discuss spam? Or have technical discussions about Trojans and viruses? Or you need to be 100% absolutely certain that email related to specific events is delivered, with no exceptions? You can't do that with Gmail / Outlook, because they have filters that you can't see and you can't control.
There are several colliding problems there (cheap cell phone plan, 2fa being via text, online purchases requiring 2fa) but it still illustrates to me the pain of doing simple stuff in the modern tech space. I wish the powers that be would work harder on solutions that don't require extra work from the people doing small, normal stuff. It would be better to have a lot more fraud occur but a lot more of the perpetrators pursued and caught. A lot of anti-fraud measures seem to be largely about passing the buck to someone else instead of actually eliminating the humans who are driving the fraud.
There's some truth in this, but I think there is a lot of room for improving things as far as making life much more painful for opportunistic assholes in general.
In the USA, I think it would be worth trying to sue Cloudflare for either "free speech" or "public nuisance" violations. Gonna reach out to the ACLU and EFF in the coming days.
Strict fraud could be handled, but everything above is really different per jurisdiction by obvious reasons. There is nothing clearly good or bad in bots, or e.g. pirates, it depends on particular cultural perception. And if one nation thinks that the action is not a crime, it doesn't make sense to them to prosecute such actions for foreign requests.
Combat fraud first so you can start to really identify the other more troublesome troublemakers.
Bots? Declare the owner. Lie about the owner? Fraud.
Crawling? Bots.
Intellectual property? That's an entire whole other industry.