Playing in the Creek

That was a well written essay with a non-sequitur AI Safety thing tacked to the end. His real world examples were concrete, and the reason to stop escalating easy to understand ("don't flood the neighbourhood by building a real dam").

The AI angle is not only even hypothetical: there is no attempt to describe or reason about a concrete "x leading to y", just "see, the same principle probably extrapolates".

There is no argument there that is sounder than "the high velocities of steam locomotives might kill you" that people made 200 years ago.

> the high velocities of steam locomotives might kill you

This obviously seems silly in hindsight. Warnings about radium watches or asbestos sound less silly, or even wise. But neither had any solid scientific studies showing clear hazard and risk. Just people being good Bayesian agents, trying to ride the middle of the exploration vs. exploitation curve.

Maybe it makes sense to spend some percentage of AI development resources on trying to understand how they work, and how they can fail.

> This [steam locomotives might kill you] obviously seems silly in hindsight.

To be fair, many people did die on level crossings and by wandering on to the tracks.

We learned over time to put in place safety fences and tunnels.