Unless you have an obvious and accessible way of getting secure third party builds whitelisted, this is still a very anti-user approach, which is not justifiable unless the user of the device isn't its owner (like with company-owned work phones).
Keep in mind that Pinephones and similar are a thing. Lots of people are hoping they don't fizzle out and die off like previous "open" phone projects. :)
Ban attestation methods that owners can't control.
I've never seen a usage of Safetynet which I would consider right, pretty much everybody thinks it creates some kind of "security" whereas it doesn't.
One very rare useful usage for it could be removing bots for game leaderboards but certainly not banking apps.
If a company wants control over devices they own, that's still fine.