We’re naturally inclined to be ok with giant distances on the horizon. It’s natural to put more emphasis on that part of the world. Hold up your thumb to the horizon and notice how many things fit alongside it compared to your thumb help downwards against the ground.
On the surface of our planet the up direction isn’t usually interesting and the down direction isn’t even there. It is therefore quite horrifying (“fun”) to imagine space going down forever.