Is it possible to allow sideloading and keep users safe?

Is it possible to let owners use their hardware as they wish, without having large companies control what they deem "safe"?

I'm not the user of my phone, I'm its owner.

Sure. But the societal losses of a vast amount of people getting scammed might in general be more important your individual wish for freedom to run anything you want on your device. I think there are important tradeoffs to be made, and that we have to acknowledge that many people in society less technically skilled might suffer from serious consequences in your proposed model of computation.

I say let them be scammed. Idiocy only grows if it's not resisted. People don't learn if they don't see the consequences. Otherwise it'll just make society head into an authoritarian socialist hellhole... not that it wasn't already going in that direction.

Yes there is a clear part from not being able to install what you want on your phone - even though you are free to buy a phone that you can - and authoritarianism. Did you know that you also can’t drive just anything on the highway and in some places you have to get your car inspected every year before you can drive it?