Tachy0n: The Last 0day Jailbreak

> The way he managed to beat a trillion dollar corporation was through the kind of simple but tedious and boring work that Apple sucks at: regression testing.

> Because, you see: this has happened before. On iOS 12, SockPuppet was one of the big exploits used by jailbreaks. It was found and reported to Apple by Ned Williamson from Project Zero, patched by Apple in iOS 12.3, and subsequently unrestricted on the Project Zero bug tracker. But against all odds, it then resurfaced on iOS 12.4, as if it had never been patched. I can only speculate that this was because Apple likely forked XNU to a separate branch for that version and had failed to apply the patch there, but this made it evident that they had no regression tests for this kind of stuff. A gap that was both easy and potentially very rewarding to fill. And indeed, after implementing regression tests for just a few known 1days, Pwn got a hit.

And now I wonder how many other projects are doing this. Is anyone running a CI farm running historical vulnerabilities on new versions of Linux/FreeBSD/OpenWRT/OpenSSH/...? It would require that someone wrote up each vulnerability in automated form (a low bar, I think), have the CI resources to throw at it (higher bar, though you could save by running a random selection on each new version), care (hopefully easy), and think of it (surprisingly hard).

Yes, regression testing--making sure bugs you've fixed don't return--is a standard part of QA. I did volunteer QA for Mozilla in college a good 20 years ago (god that number is horrifying) and they had an ever-growing suite of regression tests. Mostly for rendering/layout or JavaScript engine bugs, since part of reproducing and proving you'd fixed those was creating a minimal test case. Which you could then easily throw into the build pipeline.

Bugs are a fact of life, but burning time and money to fix them only to have them return is the worst case scenario. Organizations that care about quality are definitely investing in regression testing. Unfortunately a whole lot of orgs give QA zero respect and offshore it to the lowest bidder, if they do it at all. It's absolutely insane to me that Apple wouldn't have regression tests for jail breaks, some of the most high profile bugs in history.

You can fairly criticize Mozilla for a number of things these days. But they had a very robust QA and CI/CD setup in the early 2000s with tools like Tinderbox and Bugzilla. When DevOps came around and popularized it I was like wait, people weren't already doing this stuff??? Turned out I had been living in a bubble and that was not the norm at all.