It's also quite hard to find suitably hot rocks suitably close to the surface.
Focusing on fusion .. I think that's a legacy of 60s SF, when the fission revolution was still promising "energy too cheap to meter".
In a world where anyone could just YOLO any reactor into production with minimal red tape, consequences be damned, fission energy would actually be extremely cheap. Hence the optimism around fusion. The promise of fusion is an actualization of last century's idealistic conception of fission. It can be a silver bullet for all intents and purposes, at least once it's established with a mature supply chain.
At worst, nuclear waste contaminates a discrete section of the Earth. Climate change affects literally everywhere. The correct answer would have been to aggressively roll out fission power 40-50 years ago and then pursue renewables. You can argue that other solutions would make fission power obsolete, but we would have been in a much better spot if it'd at least been a stepping stone off fossil fuels. Instead, we have 40-50 years of shrieking and FUD from environmentalists over an issue that can be kept under control with proper regulation. The US Navy has operated reactors for over 60 years without incident, proving it can be done with proper oversight.
TL;DR nuclear has issues, but I'd take it over coal every day and twice on Sundays, at least until something better can scale.