Besides all the crappy Linux desktop software today (I have been trying multiple recent distros out on multiple new laptops... all the Linux desktop stuff now is buggy, features are gone that were there 10 years ago... it's annoying as hell). The ARM experience is one of being a second-class citizen. A ton of apps are released as AppImages or Snaps/Flatpaks. But they have to be built for both X86_64 and ARM64, and extremely few are built for the latter. Even when they are built for it, they have their own bugs to be worked around, and there's fewer users, so you wait longer for a bugfix. The end result is you have fewer choices, compatibility and support.
I love the idea of an ARM desktop. But it's going to cause fragmentation of available developer (and corporate/3rd-party) resources. ARM devices individually are even more unique than X86_64 gear is, so each one requires more integration. I'm sticking to X86_64 so I don't have to deal with another set of problems.