There's many factors in play from what your SDR white point is at, how your OS handles HDR video, what the content contains, and finally what your brain is doing.
HDR10(+) & Dolby Vision, for example, encode content at absolute luminance, so they are basically completely trash formats since that's an insane thing to expect (the spec for authoring content in this format literally just goes "lol idk do X if you think it's going to be seen in a movie theater of Y for TV and hope"). Sadly, they are also quite common. Mobile phones (both Android & iOS) are instead pushing HLG, which is better. Although then hilariously MacOS's handling of HLG was atrocious until the latest update which fixed it but only if the video contains a magic flag that iPhone sets, but isn't standard so nobody else sets it (the "avme" tag https://developer.apple.com/documentation/technotes/tn3145-h... )
There's then also just how your eyes & brain react. When HDR shows up and suddenly the white background of a page looks like a dim gray? That's 100% a perceptual illusion. The actual light being emitted didn't change, just your perception of it did. This is a very hard problem to deal with, and it's one that so far the HDR industry as a whole has basically just ignored. But it's why there's a push to artificially limit the HDR range in mixed conditions, eg https://github.com/w3c/csswg-drafts/issues/9074