Downvotes are problematic, because even the best people can't always be trusted to use them correctly. A downvote should be reserved for something offtopic, evil, mean, etc., but too often a downvote simply means "I disagree".
replies(1):
I agree the potential for damage is much higher with downvotes, but there's an asymmetry there.
The trick is to make downvotes cost a little bit of karma, IMO. And while I'm on the topic, upvotes shouldn't be as free as they are, either.
Having a downvote on stories simply accelerates the divergence of good stories and bad ones... it doesn't add any information, and in fact gives undo power to early voters. An early downvote would prevent a story from ever having a chance at the front page. (It's hard enough to get people to read the 'new' page as it is... they aren't going to read something that is five minutes old and already at -1.)
If comments could 'fall off' the page like stories can, they wouldn't need a downvote either. Also, there are fewer stories than comments, so they are easier to watch by the editors. Downvotes on comments let users help out the editors a bit.
Personally, I'd rather not have downvotes on comments either. Being able to flag trollish things is enough. If I simply don't agree, I should reply or vote up an opposing comment rather than simply downvote with no explanation.