←back to thread

Alignment is capability

(www.off-policy.com)
106 points drctnlly_crrct | 2 comments | 08 Dec 25 13:23 UTC | HN request time: 0.41s | source
Show context
js8 ◴[08 Dec 25 14:21 UTC] No.46192518[source]▶
I am not sure if this is what the article is saying, but the paperclip maximizer examples always struck me as extremely dumb (lacking intelligence), when even a child can understand that if I ask them to make paperclips they shouldn't go around and kill people.

I think superintelligence will turn out not to be a singularity, but as something with diminishing returns. They will be cool returns, just like a Brittanica set is nice to have at home, but strictly speaking, not required to your well-being.

replies(8): >>46192693 #>>46192721 #>>46192946 #>>46193471 #>>46193491 #>>46193694 #>>46193737 #>>46194236 #
1. exe34 ◴[08 Dec 25 14:37 UTC] No.46192693[source]▶
Given the kind of things Claude code does with the wrong prompt or the kind of overfitting that neural networks do at any opportunity, I'd say the paperclip maximiser is the most realistic part of AGI.

if doing something really dumb will lower the negative log likelihood, it probably will do it unless careful guardrails are in place to stop it.

a child has natural limits. if you look at the kind of mistakes that an autistic child can make by taking things literally, a super powerful entity that misunderstands "I wish they all died" might well shoot them before you realise what you said.

replies(1): >>46194407 #
2. A4ET8a8uTh0_v2 ◴[08 Dec 25 16:39 UTC] No.46194407[source]▶
Weirdly, this analogy does something for me and I am the type of person that dislikes the guardrails everywhere. There is argument to be made that a child should not be given a real bazooka to do rocket jumps or an operator with very flexible understanding of value of human life.