There Is No Antimemetics Division (2018)

The author has also written a short horror story about simulated intelligence which I highly recommend: https://qntm.org/mmacevedo

Reading mmacevedo was the only time that I actually felt dread related to AI. Excellent short story. Scarier in my opinion than the Roko's Basilisk theory that melted Yudkowsky's brain.

> Scarier in my opinion than the Roko's Basilisk theory that melted Yudkowsky's brain.

Is that correct? I thought the Roko's Basilisk post was just seen as really stupid. Agreed that "Lena" is a great, chilling story though.

It's not correct. IIRC, Eliezer was mad that someone who thought they'd discovered a memetic hazard would be foolish enough to share it, and then his response to this unintentionally invoked the Streisand Effect. He didn't think it was a serious hazard. (Something something precommit to not cooperating with acausal blackmail)

> precommit to not cooperating with acausal blackmail

He knows that can't possibly work, right? Implicitly it assumes perfect invulnerability to any method of coercion, exploitation, subversion, or suffering that can be invented by an intelligence sufficiently superhuman to have escaped its natal light cone.

There may exist forms of life in this universe for which such an assumption is safe. Humanity circa 2024 seems most unlikely to be among them.

Eliezer once told me that he thinks people aren't vegetarian because they don't think animals are sapient. And I tried to explain to him that actually most people aren't vegetarian because they don't think about it very much, and don't try to be rigorously ethical in any case, and that by far the most common response to ethical arguments is not "cows aren't sapient" but "you might be right but meat is delicious so I am going to keep eating it". I think EY is so surrounded by bright nerds that he has a hard time modeling average people.

Though in this case, in his defense, average people will never hear about Roko's Basilisk.

This shows the difference between being "bright" and being "logical". Or being "wise" vs "intelligent".

Being very good at an arbitary specific game isn't the same as being smart. Prrendit that the universe is the same as your game is not wise.