How does it pay off in the long run?
If you're competing against nothing, then I see it: it opens up a wide variety of product possibilities. But linux exists. Why not spend 1/1000th the time to adapt linux?
That's not even counting the rather substantial risk that your new OS will never approach the capabilities of linux, and may very well never become generally usable at all.
Option A: spend years and millions on a project that may never be as good as existing solutions, diverting attention and resources from actual products, or...
Option B: work on products now, using an existing, high-quality, extensible, adaptable OS, with very friendly licensing terms, for which numerous experts exist, with a proven track record of maintenance, a working plan for sustainability, a large & healthy developer community exists, etc.
It's hard to imagine how it wouldn't be a complete waste of time.