This is absolutely doable by a niche company. The problem is that you need to run this
as a business. What plagues every free/open/libre project is that they're not run as a business; so they get distracted in all different directions trying to cater to ideals about free/libre licensing and so on, and end up missing the big picture.
You need to operate this as a business first, with the freedom part being a nice bonus. Nobody cares how free your thing is if it's dead on arrival and gets beaten by an entry-level smartphone.
Make a competitive product. Nowadays that could very well just mean Android with manufacturer-sanctioned root access and preinstalled terminal & X/wayland server for those who want to run desktop apps.
The Jolla phone someone linked below actually looks like a decent product. The Android app support means it's actually usable in the modern world, and the specs look competitive.