Is it possible to allow sideloading and keep users safe?

Is it possible to let owners use their hardware as they wish, without having large companies control what they deem "safe"?

I'm not the user of my phone, I'm its owner.

Sure. But the societal losses of a vast amount of people getting scammed might in general be more important your individual wish for freedom to run anything you want on your device. I think there are important tradeoffs to be made, and that we have to acknowledge that many people in society less technically skilled might suffer from serious consequences in your proposed model of computation.

I say let them be scammed. Idiocy only grows if it's not resisted. People don't learn if they don't see the consequences. Otherwise it'll just make society head into an authoritarian socialist hellhole... not that it wasn't already going in that direction.

Yes there is a clear part from not being able to install what you want on your phone - even though you are free to buy a phone that you can - and authoritarianism. Did you know that you also can’t drive just anything on the highway and in some places you have to get your car inspected every year before you can drive it?

A key part of your analogy is "on the highway", where I am a danger to other people and public infrastructure.

I'm allowed to build a wacky unsafe DIY car and drive it around my own property without getting permission from the government. In many scenarios I don't even need a driver's license.

Bringing the analogy back around, maybe one could argue that if I let my phone get hacked such that it becomes part of a botnet or something then it is a danger to other people, but that's not the typical example. Usually these policies claim to be about protecting me from myself while using a device I own.