Thoughts on thinking

I think the article is getting at the fact that in a post-AGI world, human skill is a depreciating asset. This is terrifying because we exchange our physical and mental labor for money. Consider this: why would a company hire me if - with enough GPU and capital - they can copy-and-paste 1,000 of AI agents much smarter to do the work?

With AGI, Knowledge workers will be worth less until they are worthless.

While I'm genuinely excited about the scientific progress AGI will bring (e.g. curing all diseases), I really hope there's a place for me in the post-AGI world. Otherwise, like the potters and bakers who can't compete in the market with cold-hard industrial machines, I'll be selling my python code base on Etsy.

No Set Gauge had an excellent blog post about this. Have a read if you want a dash of existential dread for the weekend: https://www.nosetgauge.com/p/capital-agi-and-human-ambition.

This is only terrifying because of how we’ve structured society. There’s a version of the trajectory we’re on that leads to a post-scarcity society. I’m not sure we can pull that off as a species, but even if we can, it’s going to be a bumpy road.

the barrier to that version of the trajectory is that "we" haven't structured society. what structure exists, exists as a result of capital extracting as much wealth from labor as labor will allow (often by dividing class interests among labor).

agreed on the bumpy road - i don't see how we'll reach a post-scarcity society unless there is an intentional restructuring (which, many people think, would require a pretty violent paradigm shift).

That seems like a very narrow perspective. For one, it is neither clear we will end up with AGI at all—we could have reached or soon reach a plateau with the possibilities of the LLM technology—or whether it’ll work like what you’re describing; the energy requirements might not be feasible, for example, or usage is so expensive it’s just not worth applying it to every mundane task under the sun, like writing CRUD apps in Python. We know how to build flying cars, technically, but it’s just not economically sustainable to use them. And finally, you never know what niches are going to be freed up or created by the ominous AGI machines appearing on the stage.

I wouldn’t worry too much yet.

> With AGI, Knowledge workers will be worth less until they are worthless.

"Knowledge workers" being in charge is a recent idea that is, perhaps, reaching end of life. Up until WWII or so, society had more smart people than it had roles for them. For most of history, being strong and healthy, with a good voice and a strong personality, counted for more than being smart. To a considerable extent, it still does.

In the 1950s, C.P. Snow's "Two Cultures" became famous for pointing out that the smart people were on the way up.[1] They hadn't won yet; that was about two decades ahead. The triumph of the nerds took until the early 1990s.[2] The ultimate victory was, perhaps, the collapse of the Soviet Union in 1991. That was the last major power run by goons. That's celebrated in The End of History and the Last Man (1992).[3] Everything was going to be run by technocrats and experts from now on.

But it didn't last. Government by goons is back. Don't need to elaborate on that.

The glut of smart people will continue to grow. Over half of Americans with college educations work in jobs that don't require a college education. AI will accelerate that process. It doesn't require AI superintelligence to return smart people to the rabble. Just AI somewhat above the human average.

[1] https://en.wikipedia.org/wiki/The_Two_Cultures

[2] https://archive.org/details/triumph_of_the_nerds

[3] https://en.wikipedia.org/wiki/The_End_of_History_and_the_Las...

I think we think of it as 'extracting' because people are coerced into jobs that they hate. I think AI can help us exit the paradigm of working as extracting. Basically, passion economy (ai handles marketing, internet distribution). Allows you to focus on what you actually like, but it can actually make money this time.

to be trite, we've been promised a world where AI will help to alleviate the menial necessities so that we're free to pursue our passions. in reality, what we're getting is AI that replaces the human component of passion projects (art, music, engineering as craft), leaving the "actually-hard-to-replace" "low-class" roles (cashiering, trash collection, housekeeping, farming, etc) to humans who generally have few other economic options.

without a dramatic shift in wealth distribution (no less than the elimination of private wealth and the profit motive), we can't have a post-scarcity society. capitalism depends entirely upon scarcity, artificial or not.

> With AGI, Knowledge workers will be worth less until they are worthless.

The article you've linked fundamentally relies on the assumption that "the tasks can be done better/faster/cheaper by AIs". (Plus, of course, the idea that AGI would be achieved, but without this one the whole discussion would be pointless as it would lack the subject, so I'm totally fine with this one.)

Nothing about AGI (as in "a machine that can produce intelligent thoughts on a given matter") says that human and non-human knowledge workers would have some obvious leverage over each other. Just like my coworkers' existence doesn't hurt mine, a non-human intelligence is of no inherent threat. Not by definition.

Non-intelligent industrial robotics is well-researched and generally available, yet we have plenty of sweatshops because they turn out to be cheaper than robot factories. Not fun, not great, I'm no fond of this, but I'm merely taking it as a fact, as it is how it currently is. So I really wouldn't dare to unquestionably assume that "cheaper" would be true.

And then "better" isn't obvious either. Intelligence is intelligence, it can think, it can make guesses, it can make logical conclusions, and it can make mistakes too - but we've yet to see even the tiniest hints of "higher levels" of it, something that would make humans out of the league of thinking machines if we're ranking on some "quality" of thinking.

I can only buy "faster" - and even that requires an assumption that we ignore any transhumanist ideas. But, surely, "faster" alone doesn't cut it?

I’ve thought the same. Goons powered by AI, that is.