Merit is measured in imperfect ways, by other people, and fundamentally, we don't want a hierarchy of classes, even if we claim the higher rankings/elites have merited it.
Human dignity isn't contingent on outperforming others, and everyone would likely rather live somewhere that doesn't feel like constant competition is needed to enjoy leisure, food, shelter, pastimes, etc.
When it comes to who we should trust for critical work, taking decisions on our behalf, etc., we do want someone qualified. I find the idea of "qualification/qualified" much nicer than "merit". The latter seems to imply a deserved outsized reward, like it justifies not why you are given the responsibility of something important, but why you are allowed to be richer, higher ranking, etc., than others.