And that is EXACTLY the point! You can't vote it down, and that's a very good thing.
I've been friended by some people on digg - and they constantly shout battles at me: bury this, digg that. The bury brigades have gone to work on this, etc, etc.
You can't do that here, someone who hates a topic simply can not affect it. All you can do is say there are X number of people who are interested in something. It doesn't matter that some people hate the topic, or are opposed to it. They have no voice.
Example: If you have a very controversial topic that 50% of people agree with, and 50% of people disagree with. You will never see it if you have downvoting. But since there is only upvoting, you will. And I think that that is very very important, and it's one of the reasons this site is so much better than digg and reddit.
I've always thought that digg should frontpage also controversial stories, not just upvoted ones, but ones that have an almost even ratio of down/up votes.
And BTW you don't loose half of the information about votes for a topic - the algorithm here is that stories fall down over time, so time itself basically acts as a constant down vote. You have upvoting to counter that, and ignoring a topic to assist that.
Bill Gates once said something like: if I only read the stories that I am interested in, I'll never learn anything new - so I make a point of reading the entire newspaper. (I hope I remembered the quote correctly, it was from before the internet.)