It's also quite hard to find suitably hot rocks suitably close to the surface.
Focusing on fusion .. I think that's a legacy of 60s SF, when the fission revolution was still promising "energy too cheap to meter".
In a world where anyone could just YOLO any reactor into production with minimal red tape, consequences be damned, fission energy would actually be extremely cheap. Hence the optimism around fusion. The promise of fusion is an actualization of last century's idealistic conception of fission. It can be a silver bullet for all intents and purposes, at least once it's established with a mature supply chain.
At worst, nuclear waste contaminates a discrete section of the Earth. Climate change affects literally everywhere. The correct answer would have been to aggressively roll out fission power 40-50 years ago and then pursue renewables. You can argue that other solutions would make fission power obsolete, but we would have been in a much better spot if it'd at least been a stepping stone off fossil fuels. Instead, we have 40-50 years of shrieking and FUD from environmentalists over an issue that can be kept under control with proper regulation. The US Navy has operated reactors for over 60 years without incident, proving it can be done with proper oversight.
TL;DR nuclear has issues, but I'd take it over coal every day and twice on Sundays, at least until something better can scale.
I wasn't really commenting on the merits of 20th century environmentalist movements, more raising the general point that fission power has inherent costs which weren't reflected by narrow 1950s analyses of how much energy was extractable from U-235. Operation of a fission plant requires much more capex and opex than it would if we didn't care about cleanliness (waste management), security (fissile material theft prevention), or safety (meltdown prevention).
Fusion power is more complex to invent and practically depends on modern technologies that didn't exist 50 years ago, but once the first demonstration plants are operational, marginal costs to deploy and operate more should be much lower and ultimately become very low at scale.