Indeed. I started off making a general "land" for the small city a friend is working up for a prospective module of Dungeons and Dragons. I quickly found myself re-adjusting various distances, estimating the impacts of elevation, considering "rain shadows" caused by mountains, coming up with a new scale of "how far can one reasonably ride on horseback per day?" as a kind of measurement, considering climates and microclimates (then making adjustments based on trying to justify what I wanted), looking at historical patterns of settlement growth, checking in on that set of tables of population centers and occupations long ago, and so on.
If you do not consider these things, you get Monster Hotels and general ridiculousness. If it falls too close to reality, it is boring. At the same time, things can be Too Much. So, for my philosophy, you want the mountains to be taller and the valleys to be deeper ... but only sometimes. Spaces to breathe for the beleagured traveler, but then drips and splashes and slashes of Tolkien, The Black Company, and even a little whimsy to break it up.