Airbnb has a private valuation of 31 billion. Stripe has a private valuation of 20 billion. Dropbox has a public valuation (DBX) of 11 billion.
So the two most valuable companies account for about half the total value of all the YC companies. This is what a power law looks like!
So if you're building the kind of company that might be worth $100 million someday but won't ever be worth $100 billion, VCs and startup incubators might not be right for you, but just remember that a rejection from them doesn't necessarily mean you aren't on to something great.
YC, originally started as Summer Founders Program [0], was an experiment to help PG & JL allocate seed capital differently from the norm.
So the reason why "investing in competitors" makes sense for YC as a VC firm is because YC invests a tiny fraction of $1m at the earliest stage in the life of a company, similar to the dollar amounts an angel investor would commit to a fledging startup.
Angel investing dollar amounts are an order of magnitude lower than what VCs typically invest because they invest at the growth or late stage, where an idea has been shown to be viable -- at this stage -- the product matters as much as the team behind it and thus the capital requirements are higher, usually in 10x, 100x multiples of a $1m, depending on the opportunity.
IOW, VCs don't necessarily use competition as an "exclusionary criteria" as you imply, it is merely the consequence of the magnitude of the capital at risk due to the stage at which many of them choose participate in the startup lifecycle.
A VC having a portfolio of companies competiting for the same set of customers would only lead to capital erosion -- it wouldn't any make sense to allocate capital to multiple independent entities that have traction, to exploit what is essentially the same market opportunity.