Airbnb has a private valuation of 31 billion. Stripe has a private valuation of 20 billion. Dropbox has a public valuation (DBX) of 11 billion.
So the two most valuable companies account for about half the total value of all the YC companies. This is what a power law looks like!
So if you're building the kind of company that might be worth $100 million someday but won't ever be worth $100 billion, VCs and startup incubators might not be right for you, but just remember that a rejection from them doesn't necessarily mean you aren't on to something great.
This is just a power law thing and what would be expected with any large group of companies.
Success isn't linear or a bell curve, people seem to understand those two distributions rather well power law distributions rather poorly.
Yes, every group of companies is going to have some winners and some losers. But VCs and (even more so) accelerators operate in a space where the power law produces a curve that is especially steep. The vast majority of their investments will fail entirely. In order to make up for this, they need a few big winners in order to make an overall rate of return that their investors expect.
If I'm investing in large cap consumer goods company stocks, that curve is likely to look much less steep since, for example, a company like Unilever is unlikely to fail completely.
Of course, if you're not building a startup but an "ordinary", self-sustainable company (37Signals is a widely known example), then VCs and YCombinator are not for you. But that kind of follows from the definition.
Whenever you're investing in early-stage companies with low marginal costs, that deal primarily in bits, not atoms, you're likely to end up with power law outcome distributions.
You're right: if YC had a crystal ball where we could somehow only invest in the $10B kind of company and never in the $10M kind of company, we'd do that.
But I don't think such a crystal ball is possible, because companies morph too much. Famously, Microsoft's first product was an interpreter for Altair Basic, which had a total market size probably < $10M.
So when we see a startup that has an idea that seems small, we ask ourselves, "What Microsoft is this the Altair Basic of?" (http://www.paulgraham.com/altair.html). Most of the companies that today seem like moonshots started with mundane, even trivial ideas.
So if you don't currently see how your idea can become a $100B company, that doesn't mean that you won't figure it out later.
I (kind of) disagree with one part of your comment. YC can't tell whether a startup has the potential to be a $10B kind of company, but a person starting a company can certainly decide whether they want to go for a $10B company.
Many people do - a lot of startup founders are trying to be the next Google. But some of them are not, in which case YC isn't right for them (and they'd probably need to pick the right idea/product/company to work on that can succeed without needing to be a $10B company).
Btw, I think you make a great point about Microsoft/BASIC!
YC, originally started as Summer Founders Program [0], was an experiment to help PG & JL allocate seed capital differently from the norm.
So the reason why "investing in competitors" makes sense for YC as a VC firm is because YC invests a tiny fraction of $1m at the earliest stage in the life of a company, similar to the dollar amounts an angel investor would commit to a fledging startup.
Angel investing dollar amounts are an order of magnitude lower than what VCs typically invest because they invest at the growth or late stage, where an idea has been shown to be viable -- at this stage -- the product matters as much as the team behind it and thus the capital requirements are higher, usually in 10x, 100x multiples of a $1m, depending on the opportunity.
IOW, VCs don't necessarily use competition as an "exclusionary criteria" as you imply, it is merely the consequence of the magnitude of the capital at risk due to the stage at which many of them choose participate in the startup lifecycle.
A VC having a portfolio of companies competiting for the same set of customers would only lead to capital erosion -- it wouldn't any make sense to allocate capital to multiple independent entities that have traction, to exploit what is essentially the same market opportunity.