This reads like a transcript from our early design conversations! What swayed us in the end is that the upvoting and flagging mechanisms are qualitatively different. (In fact it's crazy how different they are.) Whatever's going on there is an important psychological distinction, and collapsing upvoting and flagging for technical reasons—an obvious simplicity win—loses out to that. It's also worth noting that flags are treated differently by the software than downvotes are (or would be if we had them, in the case of stories).
Once we recognized the distinction between voting and flagging it seemed more natural to have this new feature be on the flagging side, hence vouches. The jury's still out on whether it's a good idea or not, though; if it turns out bad, we might consider reverting to the voting idea.