Downvotes are problematic, because even the best people can't always be trusted to use them correctly. A downvote should be reserved for something offtopic, evil, mean, etc., but too often a downvote simply means "I disagree".
replies(1):
I agree the potential for damage is much higher with downvotes, but there's an asymmetry there.
The trick is to make downvotes cost a little bit of karma, IMO. And while I'm on the topic, upvotes shouldn't be as free as they are, either.