But if you told them about social media, I think the story would be different. Some would think it would be great, some would see it as dystopian, but neither would be right.
We don't have to imagine, though. All three of these things have captured people's imaginations since before the 50's. It's just... AI has always been closer to imagined concepts of social media more than it has been to highly advanced communication devices.
It would be utopian, like how people thought of social media in the oughts. It's a common pattern through human history. People lack the imagination to think of unintended side effects. Nuclear physics leading to nuclear weapons. Trains leading to more efficient genocide. Media distribution and printing press leading to new types of propaganda and autocracies. Oil leading to global warming. IT leading to easy surveillance. Communism leading to famine.
Some of that utopianism is wilful, created by the people with a self-interested motive in seeing that narrative become dominant. But most of it is just a lack of imagination. Policymakers taking the path of local least resistance, seeking to locally (in a temporal sense) appease, avoiding high-risk high-reward policy gambits that do not advance their local political ambitions. People being satisfied with easy just-so stories rather than humility and a recognition of the complexity and inherent uncertainty of reality.
AI, and especially ASI, will probably be the same. The material upsides are obvious. The downsides harder to imagine and more speculative. Most likely, society will be presented with a fait accompli at a future date, where once the downsides are crystallized and real, it's already too late.
Your ending sentence is certainly correct: we aren't imagining the effects of AI enough, but all of your examples are not only unconvincing, they're easy ways to ignore what downsides of AI there might be. People can easily point to how trains have done a net positive in the world and walk away from your argument thinking AI is going to do the same.
They did. I am talking about the physicists who preceded these particular physicists.
> And Communism didn't lead to famine - Soviet and Maoist policies did. Communism was immaterial to that.
The particular brand of agrarian communism and agricultural collectivization resulting from this subtype of communism did directly cause famine. The utopian revolutionaries did not predict this outcome before hand.
> People can easily point to how trains have done a net positive in the world and walk away from your argument thinking AI is going to do the same.
But that is one plausible outcome. Overall a net good, but with significant unintended consequences and high potential for misuse that is not easily predictable to people working on the technology today.
> You don't think those who worked on the Manhattan Project knew the deadly potential of the atom bomb?
Many of those who built the bomb became some of the strongest opponents. They were blinded by their passion. They were blinded by their fears. But once the bomb was built, once the bomb was dropped, it was hard to stay blind.
I say that this changed physicists, because you can't get a university degree without learning about this. They talk about the skeletons in the closet. They talk about how easy it is to fool yourself. Maybe it was the war and the power of the atom. Maybe it was the complexity of "new physics". Maybe it happened because the combination.
But what I can tell you, is that it became a very important lesson. One that no one wants to repeat:
it is not through malice, but through passion and fear that weapons of mass destruction are made.