While a space elevator doesn't contradict any fundamental limits of physics, that doesn't mean it's actually possible to build one. There is no reason to be certain that it's actually possible to create a material that has the required characteristics in terms of tensile strength to support it's own weight, plus the weight of the elevator, plus the weight of all the additional cabling. It also has to endure the huge temperature differences that it will experience along its length and from day to night and from season to season.
This is especially true considering that you don't need something that barely holds - you need something that you know will hold up to many times more weight than it needs to, so that it can be safe: the potential energy such a thing would store would be enough to dig into hundreds of meters of rock all around the world, if it ever crashed. So, you have to ensure there is no realistic chance of it ever crashing. It also has to be highly non-fragile in other ways, so that a madman with a bomb or a freak collision with an airplane or a meteor (especially likely in the thin upper layers of the atmosphere) won't bring it all down.
This combination of properties may well be completely impossible to actually achieve in a material. Even if there is no obvious basic law of physics that it would break, that doesn't mean that it wouldn't break other, harder to touch, derived laws.