Screens can't often do full brightness on the whole screen so if you come across a video or image that is supposed to have a higher contrast ratio, the system will darken everything and then brighten up the pixels that are supposed to be brighter.
Yes, there are formats that able to store a higher contrast ratio so that's why it doesn't happen on non-HDR content but the actual brightening of a portal on your screen isn't because of the format but because of your hardware (and software) choosing to interpret the format that way.
For more a practical example, if you had an 8-bit HDR image, 255 on the red channel (after inputting this number through a math function like HLG[1] to "extract" a brightness number) might mean "make this pixel really bright red" whereas 255 on a SDR format would mean "just regular red." However, each red channel is still a number between 0 and 255 on both formats but your hardware decided to make it brighter on the HDR format.
(Although in reality, HDR formats are often 10-bit or higher because 256 values is not enough range to store both color and brightness so you would see banding[2]. Also, I have been using RGB for my example but you can store color/brightness number many other ways, such as with chroma subsampling[3], especially when you realize human eyes are more sensitive to some colors more than others so you could "devote fewer bits" to some colors.)
[1] https://en.wikipedia.org/wiki/Hybrid_log%E2%80%93gamma
[2] https://en.wikipedia.org/wiki/Colour_banding
[3] https://en.wikipedia.org/wiki/Chroma_subsampling