←back to thread

265 points todsacerdoti | 1 comments | | HN request time: 0.202s | source
Show context
yjftsjthsd-h ◴[] No.44084084[source]
> The way he managed to beat a trillion dollar corporation was through the kind of simple but tedious and boring work that Apple sucks at: regression testing.

> Because, you see: this has happened before. On iOS 12, SockPuppet was one of the big exploits used by jailbreaks. It was found and reported to Apple by Ned Williamson from Project Zero, patched by Apple in iOS 12.3, and subsequently unrestricted on the Project Zero bug tracker. But against all odds, it then resurfaced on iOS 12.4, as if it had never been patched. I can only speculate that this was because Apple likely forked XNU to a separate branch for that version and had failed to apply the patch there, but this made it evident that they had no regression tests for this kind of stuff. A gap that was both easy and potentially very rewarding to fill. And indeed, after implementing regression tests for just a few known 1days, Pwn got a hit.

And now I wonder how many other projects are doing this. Is anyone running a CI farm running historical vulnerabilities on new versions of Linux/FreeBSD/OpenWRT/OpenSSH/...? It would require that someone wrote up each vulnerability in automated form (a low bar, I think), have the CI resources to throw at it (higher bar, though you could save by running a random selection on each new version), care (hopefully easy), and think of it (surprisingly hard).

replies(4): >>44084146 #>>44084820 #>>44085028 #>>44085321 #
edoceo ◴[] No.44085321[source]
There is a FOSS project I've seen but cannot remember the name of currently (beer) but I do recall their test case directory, one for each issue of merit. Thousand of them, easy. Might of been Sqlite. Something to look up to. I guess if you're not back porting fixes you'd likely not back port the tests either.
replies(1): >>44085989 #
1. TheMatten ◴[] No.44085989[source]
Glasgow Haskell Compiler project does this: https://gitlab.haskell.org/ghc/ghc/-/tree/master/testsuite/t...

Every test starting with T and a number is an example created from a corresponding issue in their tracker. And there is, well, a lot of them.