How Cloudflare blocked a monumental 7.3 Tbps DDoS attack

> sending a large volume of malicious traffic

How would an ISP determine egress is malicious? Genuinely curious.

If someone is reporting malicious traffic coming from the ISP's network then an ISP should be obligated to investigate and shut off the offending customer if necessary until they've resolved the problem.

How would this ever work at scale? These attacks come from thousands of compromised devices usually. e.g. Someone's smart fridge with 5 year old firmware gets exploited

I don't have a specific answer for that but it is really a problem that residential ISPs are going to have to solve now that gigabit or faster symmetric internet connections are becoming the norm.

All large ISPs have fancy network visibility and DDoS mitigation solutions.[1] But getting them to actually USE them for problems that aren't lighting up their monitoring dashboards is another story entirely.

(1. I know this, because I used to work for a company that made them, and the majority of worldwide ISPs were our customers.)

One simple way to do it is configure the customers routers to drop/reject all UDP/TCP packets where SRC address does not match Private IP/WAN Assigned Public IP.

Hundreds of Gbps of UDP traffic to random ports of a single destination IP from residental (?) network should be pretty easy pattern to automatically detect and throttle.

More advanced attacks are more tricky to detect, but plain dumb UDP flood should be easily detectable.

As dijit (above this comment) has noted, this is somewhat possible and automated today.

For example, one method has the attacked IP get completely null-routed, and the subsequent route is advertised. Upstream routers will pick up the null-route advertisement and drop the traffic ever closer to the source(s). The effect of the null route is that the attacked IP is unreachable by anyone until the null-route is lifted... so the aim of the DDoS isn't averted, but at least the flood of traffic won't pummel any network paths except for (ideally) the paths between the attacker(s) and the first router respecting the null-route. In my experience the DDoS tends to stop more quickly and shift away to other targets if the folks directing the attack can no longer reach the target (because: null-route) and then the null-route can be lifted sooner relative to a long-running DDoS that hasn't shifted away to other targets.

The customer's router is for the customer to configure

I cannot believe this is still not commonly done. I remember discussing this with some people in the industry over ten years ago and the sentiment was “if ISPs just stopped IP spoofing that would solve most problems”.

> Hundreds of Gbps of UDP traffic to random ports of a single destination IP from residental (?) network

You mean my legitimate QUIC file transfer?

I think ideally the customers router shouldn’t be touched, but the ISP can still do packet filtering on the next hop to drop any packets which don’t have a src ip matching the assigned WAN address of the router.

With SMTP there are services who provide a list of malicious servers so that they can be blocked at the receiving end.

I wonder if this would work in reverse, having a standardised, automated protocol that allow providers like Cloudflare to notify upstream networks of attacks in real time, so malicious traffic can be blocked closer to the source.

Genuinely curious, I'm not an expert in low-level networking ops.

Indeed, though we're at the mercy of the tyranny of the default.

Have you ever uploaded 100's of Gbps over QUIC from your residential connection to a single IP?

And the aggregate across the ISP's network could in theory be monitored - so if you were uploading 1Gbps, yes, it could be legitimate. If you and 582 others were all uploading 1Gbps to the same IP at the same time, much less likely legitimate.

> Have you ever uploaded 100's of Gbps over QUIC from your residential connection to a single IP?

Yes actually --- migration between cloud bulk storage providers.

Edit: I misread Gbps as Mbps above.

Your ISP likely knows you're part of a botnet quite early. For example many of them use magic domains as either shutoff switches or CC endpoints, so could be detected. But when was the last time anyone's ISP ever told them "hey one of your hosts is infected"?

Which residential ISP offers >100Gbps service?

It would solve a ton of other people’s problems, but cause a few for you, so it won’t be done until required by law.

E.g., customer does something stupid with addresses but the “wrong address” is something they control on another network, so it works. Egress filtering breaks it, support call and crying.

Wouldn't that need a huge amount of extra hardware to do that filtering when the routers in each customer's home are mostly idle? Just setting egress filtering as the default and letting users override that if they need to for some reason should be a good outcome. The few that do change the default hopefully know what they are doing and won't end up part of a DDoS but they'll be few anyway so the impact will still be small.

My homenet is 1GBit, so is my Internet

I.e. no traffic beyond my legitimate saturation can reach the ISP

I have saturated my link with quic or wireguard (logical or) plenty of times.

The lack of any response on high data rates would be an indicator I've only tried that once and it failed gloriously due to congestion. I don't think there's many real protocols that are unidirectional without even ACKs

The router in the customer's home cannot be trusted. With cable at least, you are able to bring in your own modem and router. Even if not, swapping it is easy, you just have to clone the original modem's MAC. In practice this is probably quite common to save money if nothing else (cable box rental is $10+/mo).

Note that spoofing source IPs is only needed by the attacker in an amplification attack, not for the amplyfing devices and not for a "direct" botnet DDOS.

I would in fact guess that it's not common at all. Setting up your own cable modem and router is going to be intimidating for the average consumer, and the ISP's answer to any problems is going to be "use our box instead" and they don't want to be on their own that way. I don't know anyone outside of people who work in IT who runs their own home router, and even many of them just prefer to let the ISP take care of it.

I think it is less common now, but ISP routers on average used to be trash with issues — bufferbloat, memory leaks, crashes — so a number of people bought a higher end router to replace the ISP provided one. Mostly tech savvy people who were not necessarily in IT.

Nowadays my ISP just uses dhcp to assign the router an address so you can plug any box into it which talks ethernet and respects dhcp leases to be a router which is nice, albiet 99.9% of people probably leave the router alone.

> Wouldn't that need a huge amount of extra hardware to do that filtering

20 years ago Cisco (probably much longer) routers were able to do this without noticeable performance overhead (ip verify unicast reverse-path). I don't think modern routers are worse. Generally filtering is expensive if you need a lot of rules which is not needed here.

Common no, very easy to proliferate though as people become aware of the savings possible. And the 2 cases I've seen where litteraly order the same model online and swap it, no configuring required. And it wasn't even the family tech support guy(me) who came up with the idea. The ISPs incuding the router as a monthly line item on the bill are litteraly indirectly asking you to do this.

Comcast/Xfinity in fact gives me a discount for using their router. Probably because (a) it lowers their support burden and (b) they are logging and selling my web traffic or at least DNS lookups.

> How would this ever work at scale?

We pay internet providers healthy amounts of money each month. Surely they can afford to hire some staff to monitor the abuse mailbox and react on it - we know they can when the MAFIAA comes knocking for copyright violations, because if they don't comply they might end up getting held liable for infractions.

That's surprising to me, it was when I used Comcast (2016) that I first purchased a cable modem. It did save me money.

Oh I also forgot that connection sharing thing they do where they broadcast a second SSID called "Xfinity WiFi" or something like that so that anyone with an Comcast login can use your connection.

> Have you ever uploaded 100's of Gbps over QUIC from your residential connection to a single IP?

I upload files to a single location, and I expect to use the max bandwidth I can whenever I do it. What's your point?

> How would this ever work at scale?

- ISP has terms of service preventing abuse,

- ISP provides an email address to receive complains about abuse

- once a ISP receives a complain, their check if a customer abused their terms of service

- once a ISP spots a customer abusing terms of service, they act upon it.

ISPs have been doing this since the time ISPs exist.

It is commonly done in a sense that probably about 50% of end users cannot spoof source IP but even if 10% (I don't know exact numbers) of end users allowed to spoof IP (due to ISP neglegence) and 1% of them are compromised (one way or another - useful software with hidden "functions" seems to be a common way) it is more than enough for a huge DDoS attack.

My point is that you don't have 100's of Gbps of bandwidth on a residential connection. In the future you might, but in the future it'll be 10's or 100's of Tbps for a large DDoS, or something.