Having to lift them via some other means first eats away almost all their advantage.
However I believe your point holds more generally for nuclear-based space propulsion. That we fear "NUKULAR!" by about two to three orders of magnitude more than is justified has kept us from having halfway decent space travel for at least a good two decades, most likely. There are a number of nuclear propulsion mechanisms that would make things like going to Mars halfway feasible instead of flights of fancy, or doing science missions in months instead of years or even decades, but people hear that you're thinking of lifting nuclear material into space and all rationality goes flying out the launch window. Nuclear is so bad that it basically reaches out through outright magic and guarantees explosions and there's no conceivable amount of preparation that could be done in people's minds to prevent the evil radiation!!!!1! from escaping and eating people's puppies.
The funny thing is that even so quite a bit of nuclear material has been lifted into space, but hearing that doesn't make people go "oh, well, maybe it's less dangerous than I thought".
I mean, I know this isn't the safest stuff in the world but I sure hope all that anti-nuclear propaganda in the 20th century actually did help prevent nuclear war because it has certainly had massively negative impacts in energy generation, environmental damage, space exploration, and who knows what else.