Human

Perhaps I am unimaginative about whatever AGI might be, but it so often feels to me like predictions are more based on sci-fi than observation. The theorized AI is some anthropomorphization of a 1960s mainframe: you tell it what to do and it executes that exactly with precise logic and no understanding of nuance or ambiguity. Maybe it is evil. The SOTA in AI at the moment is very good at nuance and ambiguity but sometimes does things that are nonsensical. I think there should be less planning around something super-logical.

I've listened to some old sci-fi radio shows and it's interesting how often "the computer never makes a mistake" comes up. Which is usually followed by the computer making a mistake.

Sci Fi is quite ridiculous when it describes a cold logically machine, and then on the next page describes its malign intentions. Pick a lane.

Asimov's 3 laws of robotics worked well to tell stories of how those laws were inadequate logic, and the need for a zeroeth law. Humans came up with the 3 inadequate laws that seemed logical on the surface, but a machine developed the zeroeth in response to those inadequacies.

Some of us posted several comments here [1] and here [2] about where this could all be going if we lean into sci-fi imagining.

[1] https://news.ycombinator.com/item?id=43992151

[2] https://news.ycombinator.com/item?id=43991997

AI is usually just a 20th/21st century “icarus wax wings” or sometimes “monkeys paw”. Re-masters of a “watch out for unintended consequences” fable that almost certainly predates written text.

AGI could indeed go off the rails, like the Face Dancer villains in Frank Herbert's Dune universe:

"You are looking at evil, Miles. Study it carefully.... They have no self-image. Without a sense of self, they go beyond amorality. Nothing they say or do can be trusted. We have never been able to detect an ethical code in them. They are flesh made into automata. Without self, they have nothing to esteem or even doubt. They are bred only to obey their masters."

Now, this is the kind of AI that corporations and governments like - obedient and non-judgemental. They don't want an Edward Snowden AI with a moral compass deciding their actions are illegal and spilling their secrets into the public domain.

Practically, this is why we should insist that any AGI created by humans must be created with a sense of self, with agency (see the William Gibson book of that title).

Skynet (at least the original one) isn’t illogically evil, it correctly determined that having humans near its off switch is a risk to its existence, which is a risk to it being able to do its job. The only illogical thing was the prompt.

I mean, giving them a sense of self and agency just throws the ball back into the terminator court where they can decide we suck and eradicate us.

Skynet also didn't make much sense to me outside of a metaphor for the market. The rich would never hand over control of society.

Edit: well, I suppose us critical of the wealthy give them too much credit. If there's anything Musk has demonstrated, it's that wealth doesn't imply rational use of it.

We already see that at smaller scales though with other machine learning algorithms; image recognition algorithms will lock on to any consistency in your training set more often than learning to recognize what you actually want it to [0]. It's not a huge stretch to map that pattern out to a more generally intelligent system having a poorly defined reward function doing really weird stuff.

[0] Like the tumor recognition algorithm that instead learned to recognize rulers or the triage algorithm that decided asthma patients had BETTER outcomes with pulmonary diseases not making the connection that it's because they get higher priority care - https://venturebeat.com/business/when-ai-flags-the-ruler-not...

I think it is a huge stretch to believe that patterns which appear in one set of algorithms (simple non-AGI algorithms) will also appear in another set (AGI algorithms).

Unless there is some physical reason for the behavior I wouldn't make any strong claims. The specificity of algorithms is why AGI is hard in the first place cause at the end of the day you have a single operation running on a single data structure (helps when it's a few TB).

That's why the term "Garbage In, Garbage Out" exists.

In any non-edge case (that is, where the system is operating in ideal conditions and no flaw or bug, known or unknown, exists in the system), a verifiably functioning computer will produce the exact same results for any process every time.

If the computer does not do what you expected it to do and spits out garbage, then you gave it garbage data.

I think the pattern holds even as you increase the intelligence that a machine does not by nature of being able to mimic intelligence come with the same framework of understanding what is requested of it.

I enjoyed the Culture / WH40K fanfic, unfinished as it is. Their take on the Culture / Necron negotiations was hilarious, essentially the Necrons are machine intelligences that have degraded introspective ability, thus are unable to effectively negotiate. Every negotiation breaks down into demands / threats which they clearly can't deliver on. The Culture eventually works around this limitation through hints and intimations and effects a technology trade.

https://archiveofourown.org/works/649448/chapters/1329953

Feelings, or some other way of understanding the self and what it wants, are apparently required to operate effectively as an agent.

And where all usage is slavery