A Note on Essential Complexity

Humans simply cannot keep the whole stack for a complex system in memory, that is why we abstract layers with APIs etc and generally specialize on one layer only.

My (Sci-Fi) book postulated that an AGI (a real AGI, after all it was Sci-Fi) would simply discard everything the humans wrote and rewrite the complete stack (including later on the Hardware and ISA) in machine code without anything unnecessary for the task and of course totally unreadable to a human. It is an interesting scenario to ponder.

OTOH a lot of the abstractions are created because problems aren't solved coherently but rather partitioned into organizational structures adding incidental complexity. The worst are <noun>-services that seem to always do too much/too little or have interface designs that suit itself than needs of clients/consumers. I spend a fair amount of time decomplexifying the org out of architectures when we realize that performance sucks because reasons.

We can do a lot of that discarding of excess by realizing that there's the data that's read, the rules applied, and the data that's written. Everything else is ephemeral and plumbing. Looking at a process that way, and seeing the amount of complexity/abstractions one can ask what all that really necessary? Does it pay for itself in some way? It has to justify its existence beyond implementing the needed process.

I question the value of frameworks sometimes[0].

[0] https://ptrthomas.wordpress.com/2006/06/06/java-call-stack-f...

While AIs are able to keep more in their “mind” at the same time than humans, there is still a cost to consider (e.g. token limit). If software requires more effort to change (adding a feature, fixing a bug) due to spaghetti architecture, then that will also add to the cost.

Secondly, we may want to keep software on a complexity level understandable by humans, in order to not become completely dependent on the software engineering AIs.

Thirdly, the same effect that we observe with software written by humans getting to complex to understand by any human, is likely to also occur with AIs of any given capacity. AIs will have to take care, just like humans have to, that the complexity doesn’t grow to exceed their abilities to maintain the software. The old adage “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.” probably also applies to AIs.

The essential complexity of software doesn’t depend on whether a human or an AI writes it. AIs may be able to handle more accidental complexity, but it’s unclear if that is an actual benefit under the cost arguments mentioned above. So maybe it will only be useful to create and maintain software with more essential complexity than humans can handle. The question is if we want to depend on such software.

I checked out your book link < https://rodyne.com/?p=1252 >, is there a preview chapter or two somewhere? That's cool you wrote a compelling sci-fi but my list is competitive :)

> there's the data that's read, the rules applied, and the data that's written

and each rule applied can do one of three basic things: https://news.ycombinator.com/item?id=32498382

I like to collapse layers that are doing the "same kind" of thing according to the taxonomy above.

I wish we would refactor our stack!

Each level throws away a lot of the effort of the preceding layer.

E.g. after heroic engineering for the OS and memory management systems to present the illusory abstraction of a flat memory address space to the runtime, it almost immediately chops it up into smaller independent "packets" that don't interact directly.

Or the whole storage <-> memory <-> slow cache <-> fast cache <-> register divide.

I think we should be building processors that are made of tens of millions of very simple processors, each with their own memory and with dedicated "network on chip"-like communication electronics. These processors would directly map to language structures like objects/structs/functions.

Essentially flattening the stack so that there is just one level between language and silicon.

I doubt the first two or three chapters will do it justice. I have put a free to download link on the bottom of that page for the complete book as an ePub, I will keep the link valid for a few days. Enjoy.

> Humans simply cannot keep the whole stack for a complex system in memory

But IMO this necessary to fully debug it, unless you can prevent all leaky abstractions.

From your page:

> I have to buy the sequel, then another, then wait, or in some cases wait forever and hope the author, or I, don’t die before finishing it!

So, how old are you -- got burned by A Song of Ice and Fire, or by The Wheel of Time? :-)

Oh yeah, and: Thanks for the download!

In my description the 'rules' are strictly "crunch code"--both "glue" and "parsley code" would be considered plumbing, with the latter being across systems.

Wow that's generous and kind of you, thanks!

Those are the ones most people who started reading fantasy in the last century remember. I started reading in 1979 and there are quite a few other good unfinished series out there.

There are also quite a few modern ones which start awesome, then fizzle out into never-ending soap operas once the author realizes they are on the gravy train and those are even more disappointing IMHO.

Books need endings :)