How Cloudflare blocked a monumental 7.3 Tbps DDoS attack

A DDoS gets some fraction of the entire internet to attack a single host.

As the internet gets more users and more devices connected, the ratio of DDoS volume to a single connections volume will only get larger.

Is there any kind of solution?

Not a 100% solution but would help greatly if ISPs:

1) performed egress filtering to prevent spoofing arbitrary source addresses

2) temporarily shut off customers that are sending a large volume of malicious traffic

Largely they do these things, it’s just not completely automatic.

> sending a large volume of malicious traffic

How would an ISP determine egress is malicious? Genuinely curious.

Apparently no solution that has gained traction, and no single solution that works everywhere. Source address filtering (BCP 38) got us part of the way, but it's difficult/undesired to do in data centers.

IoT devices (speculated to be used here) would have to have a solution upstream. Things like MUD (RFC 8520) have been proposed, but have problems too - developers need to be able to list all communications of their device and make that available somehow (MUD profile server). Some consumers will never do it on their own, and may want to prevent alerting a device manufacturer they have a device (think connected adult toy...).

Also given that IoT devices may never be updated by their owners, expect to see IoT botnet DoS attacks for years.

Capachas?

Sorry for the worst and most hated possible solution, but I thought I'd at least mention it.

Maybe too many failed capachas causes you to not connect to the IP for an hour.

If someone is reporting malicious traffic coming from the ISP's network then an ISP should be obligated to investigate and shut off the offending customer if necessary until they've resolved the problem.

How would this ever work at scale? These attacks come from thousands of compromised devices usually. e.g. Someone's smart fridge with 5 year old firmware gets exploited

How would you expect capachas to help against UDP flood? The attack works by oversaturating the network channel. Capachas is a (terribly bad) solution to prevent the server from spending CPU and transmit bandwidth on garbage request, but these wouldn't do anything if the server have too much packets receive in the first place.

Make people pay per traffic.

TIL about capachas[1]

[1]: https://en.wikipedia.org/wiki/Cachapa

I don't have a specific answer for that but it is really a problem that residential ISPs are going to have to solve now that gigabit or faster symmetric internet connections are becoming the norm.

All large ISPs have fancy network visibility and DDoS mitigation solutions.[1] But getting them to actually USE them for problems that aren't lighting up their monitoring dashboards is another story entirely.

(1. I know this, because I used to work for a company that made them, and the majority of worldwide ISPs were our customers.)

One simple way to do it is configure the customers routers to drop/reject all UDP/TCP packets where SRC address does not match Private IP/WAN Assigned Public IP.

Hundreds of Gbps of UDP traffic to random ports of a single destination IP from residental (?) network should be pretty easy pattern to automatically detect and throttle.

More advanced attacks are more tricky to detect, but plain dumb UDP flood should be easily detectable.

Consumer home/office routers provide their clients IP connectivity without reserve. Why is that the case?

The default is to allow all available bandwidth, which presumably should be the case from ISP to consumer (most likely a paid-for service), but why should that be the default at consumer router <-> IoT? What need has your printer for 500Mbps outgoing? Or my fancy toothbrush?

As dijit (above this comment) has noted, this is somewhat possible and automated today.

For example, one method has the attacked IP get completely null-routed, and the subsequent route is advertised. Upstream routers will pick up the null-route advertisement and drop the traffic ever closer to the source(s). The effect of the null route is that the attacked IP is unreachable by anyone until the null-route is lifted... so the aim of the DDoS isn't averted, but at least the flood of traffic won't pummel any network paths except for (ideally) the paths between the attacker(s) and the first router respecting the null-route. In my experience the DDoS tends to stop more quickly and shift away to other targets if the folks directing the attack can no longer reach the target (because: null-route) and then the null-route can be lifted sooner relative to a long-running DDoS that hasn't shifted away to other targets.

Locate and brick IoT devices with vulnerabilities?

Is there any method for a connected device to advertise the required throughput? Maybe some SNMP thing? That’s the only way this would work I think.

The customer's router is for the customer to configure

I cannot believe this is still not commonly done. I remember discussing this with some people in the industry over ten years ago and the sentiment was “if ISPs just stopped IP spoofing that would solve most problems”.

> Hundreds of Gbps of UDP traffic to random ports of a single destination IP from residental (?) network

You mean my legitimate QUIC file transfer?

Residential ISPs need to better police abuse of the network and they need to better respond to reports of abuse by cutting off the abusive, botnet-infected users. Of course, until there is a financial or regulatory incentive to cut off these customers, they won’t.

I think ideally the customers router shouldn’t be touched, but the ISP can still do packet filtering on the next hop to drop any packets which don’t have a src ip matching the assigned WAN address of the router.

> As the internet gets more users and more devices connected, the ratio of DDoS volume to a single connections volume will only get larger.

I'm not sure if that's the case. Large volumetric DDoS records have been increasing, but connection bandwidths have also been increasing.

7 tbps is a lot of traffic, but it only takes 7,000 nodes with 1G symetric connections to do it. Botnet sizes don't seem to be getting that much bigger.

The basic solution to volumetric DDoS is to get a bigger pipe; this works, kind of, but it's hard to get 7 Tbps of downstream capacity, and you need to be careful that you don't become a 7 Tbps reflector.

The more scalable way is using BGP to drop traffic before it gets to you. Depending on your relationship with your hosting facility and their ISPs or your ISPs, it's often pretty easy to get packet to a given IP dropped one network before yours. Ocassionally, those blocks could propagate, and things like BGP Flow Spec promise more specific filtering... dropping all packets to an attacked IP mitigages the attack for the rest of the IPs on the path, but dropping all UDP to an attacked IP might get all the attack traffic and let most non-attack traffic through... More specific rules are possible if you wanted to try to let DNS and HTTP/3 survive while being attacked.

To work against a 45 second attack, BGP based measures need a lot of automation.

We already do. Attackers use stolen capacity.

With SMTP there are services who provide a list of malicious servers so that they can be blocked at the receiving end.

I wonder if this would work in reverse, having a standardised, automated protocol that allow providers like Cloudflare to notify upstream networks of attacks in real time, so malicious traffic can be blocked closer to the source.

Genuinely curious, I'm not an expert in low-level networking ops.

But why doesn't the market do the market thing, then?

Indeed, though we're at the mercy of the tyranny of the default.

Have you ever uploaded 100's of Gbps over QUIC from your residential connection to a single IP?

And the aggregate across the ISP's network could in theory be monitored - so if you were uploading 1Gbps, yes, it could be legitimate. If you and 582 others were all uploading 1Gbps to the same IP at the same time, much less likely legitimate.

You would want the advertised speed to be approved by the user at the time of setup.

If it was automatically accepted, the malware would just change the advertisement.

> Have you ever uploaded 100's of Gbps over QUIC from your residential connection to a single IP?

Yes actually --- migration between cloud bulk storage providers.

Edit: I misread Gbps as Mbps above.

Your ISP likely knows you're part of a botnet quite early. For example many of them use magic domains as either shutoff switches or CC endpoints, so could be detected. But when was the last time anyone's ISP ever told them "hey one of your hosts is infected"?

For each separate endpoint the impact is minimal. Being part of the attack would cost you an extra $1 and you wouldn't even notice. On the other hand, ensuring the metering works correctly, reporting to the billing system works, invoicing it properly, providing support, etc. likely costs more per-customer.

Which residential ISP offers >100Gbps service?

It would solve a ton of other people’s problems, but cause a few for you, so it won’t be done until required by law.

E.g., customer does something stupid with addresses but the “wrong address” is something they control on another network, so it works. Egress filtering breaks it, support call and crying.

You don't think the proliferation of inexpensive dogshit IoT products from the Far East, running already-10-years-out-of-date versions of Linux (bonus if it has a hidden Telnet daemon with hardcoded root password!), hooked to ever-expanding 1Gbps residential fibre lines, has anything to do with it?

This represents like 75% of surveillance camera systems out there btw.

I think the increase in 1G residential connections is a bigger factor than the IoShit products. I don't think botnet node counts are getting that much bigger, but the amount of garbage each one can push certainly is.

Wouldn't that need a huge amount of extra hardware to do that filtering when the routers in each customer's home are mostly idle? Just setting egress filtering as the default and letting users override that if they need to for some reason should be a good outcome. The few that do change the default hopefully know what they are doing and won't end up part of a DDoS but they'll be few anyway so the impact will still be small.

My homenet is 1GBit, so is my Internet

I.e. no traffic beyond my legitimate saturation can reach the ISP

I have saturated my link with quic or wireguard (logical or) plenty of times.

The lack of any response on high data rates would be an indicator I've only tried that once and it failed gloriously due to congestion. I don't think there's many real protocols that are unidirectional without even ACKs

The router in the customer's home cannot be trusted. With cable at least, you are able to bring in your own modem and router. Even if not, swapping it is easy, you just have to clone the original modem's MAC. In practice this is probably quite common to save money if nothing else (cable box rental is $10+/mo).

Note that spoofing source IPs is only needed by the attacker in an amplification attack, not for the amplyfing devices and not for a "direct" botnet DDOS.

Banks have already figured out fraud detection through pattern recognition, ISPs can do the same. When a connection has never used more than 300/10 of a 1000/1000 link and 80% of that was TCP with dstport 80 or 443, then it starts doing /900 UDP to every possible dstport, maybe something is wrong?

"Your network is generating an extraordinary amout of traffic, which is likely the result of a virus-infected device. As a result, we have lowered your speed to 100/20. Please read the steps to check your devices and unlock your connection here: ____"

Banks have way lower traffic and slower reaction times than what cf needs to support.

Lowering the speed means "good" traffic is also impacted, resulting in higher timeouts.

count the number of events isn't cheap either.

I would in fact guess that it's not common at all. Setting up your own cable modem and router is going to be intimidating for the average consumer, and the ISP's answer to any problems is going to be "use our box instead" and they don't want to be on their own that way. I don't know anyone outside of people who work in IT who runs their own home router, and even many of them just prefer to let the ISP take care of it.

IoS botnets depend on total number of devices and not individual bandwidth. Most IoT devices have cheap network chipsets and unoptimized networking stacks, I wouldn't expect them to saturate a 100mbps connection.

I think it is less common now, but ISP routers on average used to be trash with issues — bufferbloat, memory leaks, crashes — so a number of people bought a higher end router to replace the ISP provided one. Mostly tech savvy people who were not necessarily in IT.

Nowadays my ISP just uses dhcp to assign the router an address so you can plug any box into it which talks ethernet and respects dhcp leases to be a router which is nice, albiet 99.9% of people probably leave the router alone.

Good idea. People only learn that something is wrong, when... they don't have internet anymore ;D.

> Wouldn't that need a huge amount of extra hardware to do that filtering

20 years ago Cisco (probably much longer) routers were able to do this without noticeable performance overhead (ip verify unicast reverse-path). I don't think modern routers are worse. Generally filtering is expensive if you need a lot of rules which is not needed here.

Common no, very easy to proliferate though as people become aware of the savings possible. And the 2 cases I've seen where litteraly order the same model online and swap it, no configuring required. And it wasn't even the family tech support guy(me) who came up with the idea. The ISPs incuding the router as a monthly line item on the bill are litteraly indirectly asking you to do this.

Comcast/Xfinity in fact gives me a discount for using their router. Probably because (a) it lowers their support burden and (b) they are logging and selling my web traffic or at least DNS lookups.

So many false positives can happen here.

Most ISPs are already a pain in the ass to deal with. (Fuck you Charter/Spectrum). I don’t trust them to do their due diligence and implement this correctly. Or worse, abuse it.

“hey you pay for 1000/300 package. We detected abnormal traffic. Now you get throttled to 100/100. But still pay 1000/30”. Then they will drag on the resolution process until you give up.

Capacha =/= Cachapa =/= CAPTCHA

> How would this ever work at scale?

We pay internet providers healthy amounts of money each month. Surely they can afford to hire some staff to monitor the abuse mailbox and react on it - we know they can when the MAFIAA comes knocking for copyright violations, because if they don't comply they might end up getting held liable for infractions.

Economic fraud detection is like trying to find a needle in a haystack.

Blocking DDoS is like trying to separate the shit from the bread in a shit sandwich.

It's a completely different problem.

That's surprising to me, it was when I used Comcast (2016) that I first purchased a cable modem. It did save me money.

Users don't want to manage it, and ISPs don't want the tickets.

Heuristic based systems would probably work in most homes, where devices are limited by their historical bandwidth. New devices are unthrottled, existing devices are limited by their historical bandwidth usage with some bursting.

I think most ISPs have apps to control your router now, you could have it trigger a push notification like "Device X is using more bandwidth than normal, and we're throttling it. Press SCARY BUTTON to unthrottle."

Oh I also forgot that connection sharing thing they do where they broadcast a second SSID called "Xfinity WiFi" or something like that so that anyone with an Comcast login can use your connection.

> Have you ever uploaded 100's of Gbps over QUIC from your residential connection to a single IP?

I upload files to a single location, and I expect to use the max bandwidth I can whenever I do it. What's your point?

> How would this ever work at scale?

- ISP has terms of service preventing abuse,

- ISP provides an email address to receive complains about abuse

- once a ISP receives a complain, their check if a customer abused their terms of service

- once a ISP spots a customer abusing terms of service, they act upon it.

ISPs have been doing this since the time ISPs exist.

> Source address filtering (BCP 38) got us part of the way, but it's difficult/undesired to do in data centers.

BCP 38 is applicable in the DC environment, especially between an operator (hosting/cloud provider) and the customer. Where it is from hard to not practical to use is the network backbone and link between different ISPs. But that's would be a minor problem if BCP 38 will be applied to all stub networks.

It is commonly done in a sense that probably about 50% of end users cannot spoof source IP but even if 10% (I don't know exact numbers) of end users allowed to spoof IP (due to ISP neglegence) and 1% of them are compromised (one way or another - useful software with hidden "functions" seems to be a common way) it is more than enough for a huge DDoS attack.

My point is that you don't have 100's of Gbps of bandwidth on a residential connection. In the future you might, but in the future it'll be 10's or 100's of Tbps for a large DDoS, or something.