Being annoyed, inconvenienced, or even negatively impacted by the speech acts of others is by design. To throw that out is to make a calculation that without freedom of speech, your perspective will be the natural default without activism to upset it. A dangerous assumption.
You have that right. But doing this is not always wise. Labeling people as immoral and ostracizing them, especially on 50/50 issues, is one of the reason why the American political system is so radicalized at the moment.
In his Second Inaugural, Lincoln threads the needle in a way that is frankly unachievable for even most skilled politicians. "Both read the same Bible and pray to the same God and each invokes His aid against the other" seems like an acknowledgement of moral nuance, but he follows it up with, "It may seem strange that any men should dare to ask a just God's assistance in wringing their bread from the sweat of other men's faces but let us judge not that we be not judged."
Speaking to a nation in which a part of it is in open revolt over the right to keep other humans as slaves is certainly an extreme case. But it isn't categorically different from any other political struggle. People are going to accuse one another of being immoral. It's the human condition. A legal system that protects this behavior is the bedrock of democracy. It doesn't matter how annoying you find the people doing the judging.