I preferred when (way back when) it showed the actual vote totals (or up/down numbers, I don't quite remember). It also greyed posts btw. At some point they changed it, the justification was something about vote gaming/brigading and spam, which was reasonable but it wasn't a crucial fix to a very critical problem either (IMHO).
Most importantly, this change was made before dang became moderator, to try and fix certain ways the community worked, and nearly all of those ways have changed a lot because of the changes in moderation, so I wouldn't mind seeing that decision to be revisited. I doubt that'll happen though, it's just too safe and easy not to show vote numbers.
As for actual greying, I dislike the fact that it makes reading harder very much. If some people have downvoted it, doesn't mean I don't want to read it. For the same reason I have 'showdead' on in my profile settings (I really wonder how many people have this btw), also because (as I have explained in older comments of mine) I am very much against the concept of hellbanning as it is applied on HN. Again only last week I saw a few 'dead' accounts posting just reasonable (not super-informative or upvote-worthy, but just "fine") comments. Checking their post history I could see some "bad" comments that surely deserved all the downvotes they got, that probably triggered the hellban, but that doesn't excuse in any way to waste these people's time like that, writing reasonable comments, thinking they are participating in conversation, unknowing that the large majority of users won't even see their posts. There's no justification for that, if they need some kind of lasting sanction, have them be notified about it, maybe a posting timeout or something.
What would be better if comments with a certain number of downvotes would auto-collapse, like Reddit does. Even better, adjust the threshold per subthread so the total number of visible direct replies is between 10 and 15, or some number. I'd have locally scripted this myself a long time ago if the downvote-numbers were available.
However, you also seem to have a problem with the greying that it's signalling "unpopular opinion!" to other users, if I understand your remark about "stigma" correctly. Meaning that even if it didn't hamper legibility, if they used a different colour or perhaps a sad smiley in various states of desperation (I dunno), you'd still find a problem with that? Cause I'm not very concerned about that at all. IMO, the signal is the message, like say your posts in this thread, came off to me as somewhat whiny, regardless of the actual opinions in your post history that you were complaining about getting downvotes (which I still haven't looked at, btw, but I'm not sure I would want to), that kind of tone signals a lot stronger to me, downvotes or no, grey or no, opinion that I agree with or no. But maybe once you manage to take downvotes less personally, this particular side of the issue will become less of a concern for you as well.