Last time I read up on OLED in VR, it was said that pancake lenses dissipate too much light. Might be dated of course, and iirc there is now at least one OLED+pancake HMD on the market.
I have the Bigscreen Beyond 2 which is OLED + pancake fine. But only if you have the perfect light seal that the BSB face gasket ensures. Your eyes just adjust to it and I never thought about it while using it. The upside of having perfect blacks is sooooo worth it in my opinion. Flight sims in VR at night are an amazing experience