Language is not essential for the cognitive processes that underlie thought

This is an important result.

The actual paper [1] says that functional MRI (which is measuring which parts of the brain are active by sensing blood flow) indicates that different brain hardware is used for non-language and language functions. This has been suspected for years, but now there's an experimental result.

What this tells us for AI is that we need something else besides LLMs. It's not clear what that something else is. But, as the paper mentions, the low-end mammals and the corvids lack language but have some substantial problem-solving capability. That's seen down at squirrel and crow size, where the brains are tiny. So if someone figures out to do this, it will probably take less hardware than an LLM.

This is the next big piece we need for AI. No idea how to do this, but it's the right question to work on.

[1] https://www.nature.com/articles/s41586-024-07522-w.epdf?shar...

Brain size isn't necessarily a very good correlate of intelligence. For example dolphins and elephants have bigger brains than humans, and sperm whales have much bigger brains (5x by volume). Neanderthals also had bigger brains than modern humans, but are not thought to have been more intelligent.

A crow has a small brain, but also has very small neurons, so ends up having 1.5B neurons, similar to a dog or some monkeys.

Not sure neuron number correlates to smarts, either.

https://www.scientificamerican.com/article/gut-second-brain/

There are 100 million in my gut, but it doesn't solve any problems that aren't about poop, as far as I know.

https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n...

If the suspiciously round number is accurate, this puts the human gut somewhere between a golden hamster and ansell's mole-rat, and about level with a short-palated fruit bat.

> What this tells us for AI is that we need something else besides LLMs.

Basically we need Multimodal LLM's (terrible naming as it's not an LLM then but still).

Agreed. It's architecture that matters, although for a given brain architecture (e.g. species) there might be benefits to scale. mega-brain vs pea-brain.

I was just pointing out that a crow's brain is built on a more advanced process node than our own. Smaller transistors.

That makes sense. Birds are very weight-limited, so there's evolutionary pressure to keep the mass of the control system down.

I don't know what we need. Nor does anybody else, yet. But we know what it has to do. Basically what a small mammal or a corvid does.

There's been progress. Look at this 2020 work on neural net controlled drone acrobatics.[1] That's going in the right direction.

[1] https://rpg.ifi.uzh.ch/docs/RSS20_Kaufmann.pdf

You could say language is just the "communication module" but there has got to be another whole underlying interface where non-verbal thoughts are modulated/demodulated to conform to the language expected to be used when communication may or may not be on the agenda.

I suspect there is more going on with your gut neurons then you would expect. If nothing else, the vagus nerve I had to direct communication link.

I like to think that it is my gut brain that is telling me that it's okay to have that ice cream...

Is it important? To who? Anyone with half a brain is aware that language isn't the only way to think. I can think my way through all kinds of things in 3-d space without a single word uttered in any internal monologue and I'm not remotely unique - this kind of thing is put in all kinds of math and iq'ish like tests one takes as a child.

> What this tells us for AI is that we need something else besides LLMs

Not to over-hype LLMs, but I don't see why this results says this. AI doesn't need to do things the same way as evolved intelligence has.

Well said! This is a great restatement of the core setup of the Chomskian “Generative Grammar” school, and I think it’s an undeniably productive one. I haven’t read this researchers full paper, but I would be sad (tho not shocked…) if it didn’t cite Chomsky up front. Beyond your specific point re:interfaces—which I recommend the OG Syntactic Structures for more commentary on—he’s been saying what she’s saying here for about half a century. He’s too humble/empirical to ever say it without qualifiers, but IMO the truth is clear when viewed holistically: language is a byproduct of hierarchical thought, not the progenitor.

This (awesome!) researcher would likely disagree with what I’ve just said based on this early reference:

  In the early 2000s I really was drawn to the hypothesis that maybe humans have some special machinery that is especially well suited for computing hierarchical structures.

…with the implication that they’re not, actually. But I think that’s an absurd overcorrection for anthropological bias — humans are uniquely capable of a whole host of tasks, and the gradation is clearly a qualitative one. No ape has ever asked a question, just like no plant has ever conceptualized a goal, and no rock has ever computed indirect reactions to stimuli.

LLM as a term is becoming quite broad; a multi-modal transformer-based model with function calling / ReAct finetuning still gets called an LLM, but this scaffolding might be all that’s needed.

I’d be extremely surprised if AI recapitulates the same developmental path as humans did; evolution vs. next-token prediction on an existing corpus are completely different objective functions and loss landscapes.

To a point. If you drill down this far into the fundamentals of cognition you begin to define it. Otherwise you may as well call a cantaloupe sentient

I don't think anyone defines AI as "doing the thing that biological brains do" though, we define it in terms of capabilities of the system.

It doesn't need to, but evolved intelligence is the only intelligence we know of.

Similar reason we look for markers of Earth-based life on alien planets: it's the only example we've got of it existing.

Chomsky is shockingly unhumble. I admire him but he's a jerk who treats people who disagree with him with contempt. It's fun to read him doing this but it's uncollegiate (to say the least).

Also, calling "generative grammar" productive seems wrong to me. It's been around for half a century -- what tools has it produced? At some point theory needs to come into contact with empirical reality. As far as I know, generative grammar has just never gotten to this point.

Lol, it’s insane how some people will track everything back to AI

Who has he mistreated?

Before you say things this patiently dumb you should probably wonder what question the researchers are actually interested in and why your average experience isn't sufficient proof.

I am 3-d rotating this comment in my head right now

My first thought as well - “AGI via LLM” implies that our grey matter is merely a substrate for executing language tasks: just swap out bio-neurons for a few H100s and viola, super intelligence.

I asked both OpenAI and Claude the same difficult programming question. Each gave a nearly identical response down to the variable names and example values.

I then looked it up and they had each copy/pasted the same Stack overflow answer.

Furthermore, the answer was extremely wrong, the language I used was superficially similar to the source material, but the programming concepts were entirely different.

What this tells me is there is clearly no “reasoning” happening whatsoever with either model, despite marketing claiming as such.

Ok, but at least it suggests that this other thing might be more efficient in some ways.

One reason might that LLMs are successful because of the architecture, but also, just as importantly because they can be trained over a volume and diversity of human thought that’s encapsulated in language (that is on the internet). Where are we going to find the equivalent data set that will train this other kind of thinking?

Open AI O1 seems to be trained on mostly synthetic data, but it makes intuitive sense that LLMs work so well because we had the data lying around already.

In these discussions, I always knee-jerk into thinking "why don't they just look inward on their own minds". But the truth is, most people don't have much to gaze upon internally... they're the meat equivalent of an LLM that can sort of sound like it makes sense. These are the people always bragging about how they have an "internal monologue" and that those that don't are aliens or psychotics or something.

The only reason humans have that "communication model" is because that's how you model other humans you speak to. It's a faculty for rehearsing what you're going to say to other people, and how they'll respond to it. If you have any profound thoughts at all, you find that your spoken language is deficient to even transcribe your thoughts, some "mental tokens" have no short phrases that even describe them.

The only real thoughts you have are non-verbal. You can see this sometimes in stupid schoolchildren who have learned all the correct words to regurgitate, but those never really clicked for them. The mildly clever teachers always assume that if they thoroughly practice the terminology, it will eventually be linked with the concepts themselves and they'll have fully learned it. What's really happening is that there's not enough mental machinery underneath for those words to ever be anything to link up with.

I'm not convinced the result is as important here as the methods. Separating language from complex cognition when evaluating individuals is difficult. But many of the people I've met in neuroscience that study language and cognitive processes do not hold the opinion that one is absolutely reliant on the other in all cases. It may have been a strong argument a while ago, but everytime I've seen a presentation on this relationship it's been to emphasize the influence culture and language inevitably have on how we think about things. I'm sure some people believe that one cannot have complex thoughts without language, but most people in speech neuro I've met in language processing research find the idea ridiculous enough they wouldn't bother spending a few years on that kind of project just to disapprove a theory.

On the other hand, further understanding how to engage complex cognitive processes in nonverbal individuals is extremely useful and difficult to accomplish.

I think you may underestimate what these models do.

Proper multimodal models natively consider whatever input you give them, store the useful information in an abstracted form (i.e not just text), building it's world model, and then output in whatever format you want it to. It's no different to a mammals, just the inputs are perhaps different. Instead of relying on senses, they rely on text, video, images and sound.

In theory you could connect it to a robot and it could gather real world data much like a human, but would potentially be limited to the number of sensors/nerves it has. (on the plus side it has access to all recorded data and much faster read/write than a human).

This view represents one possible subjective experience of the world. But there are many different possible ways a human brain can learn to experience the world.

I am a sensoral thinker, I often think and internally express myself in purely images or sounds. There are, however, some kinds of thoughts I've learned I can only fully engage with if I speak to myself out loud or at least inside of my head.

The most appropriate mode of thought depends upon the task at hand. People don't typically brag about having internal monologues. They're just sharing their own subjective internal experience, which is no less valid than a chiefly nonverbal one.

I think one big problem is that people understand LLMs as text-generation models, when really they're just sequence prediction models, which is a highly versatile, but data-hungry, architecture for encoding relationships and knowledge. LLMs are tuned for text input and output, but they just work on numbers and the general transformer architecture is highly generalizable.

> What this tells us for AI is that we need something else besides LLMs

You mean besides a few layers of LLMs near input and output that deal with tokens? We have the rest of the layers.

Those "few layers" sum up all of linguistics.

1. Syntax

2. Semantics

3. Pragmatics

4. Semiotics

These are the layers you need to solve.

Saussure already pointed out these issues over a century ago, and Linguists turned ML Researchers like Stuart Russell and Paul Smolensky tried in vain to resolve this.

It basically took 60 years just to crack syntax at scale, and the other layers are still fairly far away.

Furthermore, Syntax is not a solved problem yet in most languages.

Try communicating with GPT-4o in colloquial Bhojpuri, Koshur, or Dogri, let alone much less represented languages and dialects.

Don’t assume whales are less intelligent than humans. They’re tuned for their environment. They won’t assemble machines with their flippers but let’s toss you naked in the pacific and see if you can communicate and collaborate with peers 200km away on complex hunting strategies.

What was the question?

As far as I understand it, it's just output and speaking is just enclosed in tags, that the body can act on, much like inline code output from an LLM.

e.g. the neural electrochemical output has a specific sequence that triggers the production of a certain hormone in your pituitary gland for e.g. and the hormone travels to the relevant body function activating/stopping it.

They don’t wonder. They’d happily produce entire novels of (garbage) text if trained on gibberish. They wouldn’t be confused. They wouldn’t hope to puzzle out the meaning. There is none, and they work just fine anyway. Same for real language. There’s no meaning, to them (there’s not really a “to” either).

The most interesting thing about LLMs is probably how much relational information turns out to be encoded in large bodies of our writing, in ways that fancy statistical methods can access. LLMs aren’t thinking, or even in the same ballpark as thinking.

I think the data is way more important for the success of LLMs than the architecture although I do think there's something important in the GPT architecture in particular. See this talk for why: [1]

Warning, watch out for waving hands: The way I see it is that cognition involves forming an abstract representation of the world and then reasoning about that representation. It seems obvious that non-human animals do this without language. So it seems likely that humans do too and then language is layered on top as a turbo boost. However, it also seems plausible that you could build an abstract representation of the world through studying a vast amount of human language and that'll be a good approximation of the real-world too and furthermore it seems possible that reasoning about that abstract representation can take place in the depths of the layers of a large transformer. So it's not clear to me that we're limited by the data we have or necessarily need a different type of data to build a general AI although that'll likely help build a better world model. It's also not clear that an LLM is incapable of the type of reasoning that animals apply to their abstract world representations.

[1] https://youtu.be/yBL7J0kgldU?si=38Jjw_dgxCxhiu7R

*patently

in the high entropy world we have, we are forced to assume that the first thing that arises as a stable pattern is inevitably the most likely, and the most likely to work. there is no other pragmatic conclusion to draw.

for more, see "Assembly Theory"

Videos are a rich set of non verbal data that could be used to train AIs.

Feed it all the video ever recorded, hook it up to web cams, telescopes, etc. This says a lot about how the universe works, without using a single word.

Title doesn't mean bullet trains can't fly, but do imply what call flights could be more than moving fast, and effects of wings might be worth discussing.

When you look at how humans play chess they employ several different cognitive strategies. Memorization, calculation, strategic thinking, heuristics, and learned experience.

When the first chess engines came out they only employed one of these: calculation. It wasn't until relatively recently that we had computer programs that could perform all of them. But it turns out that if you scale that up with enough compute you can achieve superhuman results with calculation alone.

It's not clear to me that LLMs sufficiently scaled won't achieve superhuman performance on general cognitive tasks even if there are things humans do which they can't.

The other thing I'd point out is that all language is essentially synthetic training data. Humans invented language as a way to transfer their internal thought processes to other humans. It makes sense that the process of thinking and the process of translating those thoughts into and out of language would be distinct.

> What this tells us for AI is that we need something else besides LLMs.

Humans not taking this approach doesn’t mean that AI cannot.

Had to do with connection pooling.

Humans copy/paste from SO too. Does that prove humans can’t reason?

> It's not clear to me that LLMs sufficiently scaled won't achieve superhuman performance on general cognitive tasks

If "general cognitive tasks" means "I give you a prompt in some form, and you give me an incredible response of some form " (forms may differ or be the same) then it is hard to disagree with you.

But if by "general cognitive task" you mean "all the cognitive things that human do", then it is really hard to see why you would have any confidence that LLMs have any hope of achieving superhuman performance at these things.

>Does that prove humans can’t reason?

It could be said not as well as the ones that don't need SO.

It sounds like you think this research is wrong? (it claims llms can not reason)

https://arstechnica.com/ai/2024/10/llms-cant-perform-genuine...

or do you maybe think no logical reasoning is needed to do everything a human can do? Tho humans seem to be able to do logical reasoning

It says "current" LLMs can't "genuinely" reason. Also, one of the researchers then posted an internship for someone to work on LLM reasoning.

I think the paper should've included controls, because we don't know how strong the result is. They certainly may have proven that humans can't reason either.

I think if you gave it the same biological inputs as a biological brain you would quickly see the lack of capabilities in any man made system.

Not only that but also LLMs "think" in a latent representation that is several layers deep. Sure, the first and last layers make it look like it is doing token wrangling, but what is happening in the middle layers is mostly a mystery. First layer deals directly with the tokens because that is the data we are observing (a "shadow" of the world) and last layer also deals with tokens because we want to understand what the network is "thinking" so it is a human specific lossy decoder (we can and do remove that translator and plug the latent representations to other networks to train them in tandem). There is no reason to believe that the other layers are "thinking in language".

I believe what this tells is that thought requires blood flow in the brain of mammals.

Stepping back a level, it may only actually tell us that MRIs measure blood flow.

Let's toss a whale on land and see if it can communicate and collaborate with peers 10 ft away on anything. I don't think being tuned to communicate underwater makes them more intelligent than humans.

For those interested in the history, this is in fact the Neural Network research path that predated LLMs. Not just in the sense that Hinton et al and the core of the "Parallel Distributed Processing"/Connectionist school were always opposed to Chomsky's identification of brain-thought-language, but that the original early 2000s NSF grant awarded to Werbos, Ng, LeCun et al was for "Deep Learning in the Mammalian Visual Cortex." In their research program, mouse intelligence was posited as the first major challenge.

It's "patently" and maybe understand the definition of "average" before using it.

Once you've figured out how to use language, explain why this is important and to who. Then maybe what the upshot will be. The fact that someone has proven something to be true doesn't make it important.

The comment I replied to made it sound like it's important to the field of AI. It is not. Almost zero serious researchers think LLMs all by themselves are "enough". People are working on all manner of models and systems incorporating all kinds of things "not LLM". Practically no one who actually works in AI reads this paper and changes anything, because it only proves something they already believed to be true and act accordingly.

It’s impossible to overstate how crude and useless blood flow MRI studies are, at least relative to the hype they receive.

Spoiler alert: brains require a lot of blood, constantly, just to not die. Looking at blood flow on an MRI to determine neural circuitry has to deal with the double whammy of both an extremely crude tool and a correlation/causation fallacy.

This article and the study are arguably useless.

> One reason might that LLMs are successful because of the architecture, but also, just as importantly because they can be trained over a volume and diversity of human thought that’s encapsulated in language (that is on the internet). Where are we going to find the equivalent data set that will train this other kind of thinking?

Probably by putting simulated animals into simulated environments where they have to survive and thrive.

Working at animal level is uncool, but necessary for progress. I had this argument with Rod Brooks a few decades back. He had some good artificial insects, and wanted to immediately jump to human level, with a project called Cog.[1] I asked him why he didn't go for mouse level AI next. He said "Because I don't want to go down in history as the inventor of the world's greatest artificial mouse."

Cog was a dud, and Brooks goes down in history as the inventor of the world's first good robotic vacuum cleaner.

[1] https://en.wikipedia.org/wiki/Cog_(project)

The projects mapping the brain, combined with research on what areas do, should tell us what components are necessary for our design. Studying the behavior of their specialist structures will tell us how to make purpose-built components for these tasks. Even if not, just attempting to split up the global behavior in that many ways with specialist architecture might help. We can also imitate how the components synchronize together, too.

An example was the problem of memory shared between systems. ML people started doing LLM’s with RAG. I looked into neuroscience which suggested we need a hippocampus model. I found several papers with hippocampus-like models. Combining LLM’s, vision, etc with hippocampus-like model might get better results. Rinse repeat for these other brain areas wherever we can understand them.

I also agree on testing the architectures with small, animal brains. Many do impressive behaviors that we should be able to recreate in simulators or with robotics. Some are useful, too, like how geese are good at security. Maybe embed a trained, goose brain into a camera system.

If they had human controls, they might well show that some humans can’t do any better, but based on how they generated test cases, it seems unlikely to me that doing so would prove that humans cannot reason (of course, if that’s actually the case, we cannot trust ourselves to devise, execute and interpret these tests in the first place!)

Some people will use any limitation of LLMs to deny there is anything to see here, while others will call this ‘moving the goalposts’, but the most interesting questions, I believe, involve figuring out what the differences are, putting aside the question of whether LLMs are or are not AGIs.

> It's not clear to me that LLMs sufficiently scaled won't achieve superhuman performance

To some extent this is true.

To calculate A + B you could for example generate A, B for trillions of combinations and encode that within the network. And it would calculate this faster than any human could.

But that's not intelligence. And Apple's research showed that LLMs are simply inferring relationships based on the tokens it has access to. Which you can throw off by adding useless information or trying to abstract A + B.

I always start with God’s design thinking it is best. That’s our diverse, mixed-signal, brain architecture followed by a good upbringing. That means we need to train brain-like architectures in the same way we train children. So, we’ll need whatever data they needed. Multiple streams for different upbringings, too.

The data itself will be most senses collecting raw data about the world most of the day for 18 years. It might require a camera on the kid’s head which I don’t like. I think people letting a team record their life is more likely. Split the project up among many families running in parallel, 1-4 per grade/year. It would probably cost a few million a year.

(Note: Parent changes might require an integration step during AI training or showing different ones in the early years.)

The training system would rapidly scan this information in. It might not be faster than human brains. If it is, we can create them quickly. That’s the passive learning part, though.

Human training involves asking lots of questions based on internal data, random exploration (esp play) with reinforcement, introspection/meditation, and so on. Self-driven, generative activities whose outputs become inputs into the brain system. This training regiment will probably need periodic breaks from passive learning to ask questions or play which requires human supervision.

Enough of this will probably produce… disobedient, unpredictable children. ;) Eventually, we’ll learn how to do AI parenting where the offspring are well-behaved, effective servants. Those will be fine-tuned for practical applications. Later, many more will come online which are trained by different streams of life experience, schooling methods, etc.

That was my theory. I still don’t like recording people’s lives to train AI’s. I just thought it was the only way to build brain-like AI’s and likely to happen (see Twitch).

My LLM concept was to do the same thing with K-12 education resources, stories, kids games, etc. Parents already could tell us exactly what to use to gradually build them up since they did that for their kids year by year. Then, several career tracts layering different college books and skill areas. I think it would be cheaper than GPT-4 with good performance.

The later.

While I generally do suspect that we need to invent some new technique in the realm of AI in order for software to do everything a human can do, I use analogies like chess engines to caution myself from certainty.

It's probably more relevant to compare intraspecies rather than interspecies.

And it turns out that human brain volume and intelligence are moderately-highly correlated [1][2]!

[1]: https://pmc.ncbi.nlm.nih.gov/articles/PMC7440690/ [2]: https://www.sciencedirect.com/science/article/abs/pii/S01602...

> What this tells us for AI is that we need something else besides LLMs.

Perhaps, but the relative success of trained LLMs acting with apparent generalised understanding may indicate that it is simply the interface that is really an LLM post training;

That the deeper into the network you go (the further from the linguistic context), the less things become about words and linguist structure specifically and the more it becomes about things and relations in general.

(This also means that multiple interfaces can be integrated, sometimes making translation possible, e.g.: image <=> tree<string>)

Nobody, people are just crying because Chomsky calls them out, rationally, on their intellectual and/or political bullshit, and this behavior is known as projection.

I’ll pop in with a friendly “that research is definitely wrong”. If they want to prove that LLMs can’t reason, shouldn’t they stringently define that word somewhere in their paper? As it stands, they’re proving something small (some of today’s LLMs have XYZ weaknesses) and claiming something big (humans have an ineffable calculator-soul).

LLMs absolutely 100% can reason, if we take the dictionary definition; it’s trivial to show their ability to answer non-memorized questions, and the only way to do that is some sort of reasoning. I personally don’t think they’re the most efficient tool for deliberative derivation of concepts, but I also think any sort of categorical prohibition is anti-scientific. What is the brain other than a neural network?

Even if we accept the most fringe, anthropocentric theories like Penrose & Hammerhoff’s quantum tubules, that’s just a neural network with fancy weights. How could we possibly hope to forbid digital recreations of our brains from “truly” or “really” mimicking them?

Linguistics is not living! Language does not capture reality! So no matter how much you solve you're no closer to AGI

Even in cognitive tasks expressed via language, something like a memory feels necessary. At which point it’s not a LLM as in a generic language model. It would become a language model conditioned on the memory state.

If you don’t read or understand the code, then no, you aren’t reasoning.

The condition of “some people are bad at thing” does not equal “computer better at thing than people”, but I see this argument all the time in LLM/AI discourse.

> To calculate A + B you could for example generate A, B for trillions of combinations and encode that within the network. And it would calculate this faster than any human could.

I don't feel like this is a very meaningful argument because if you can do that generation then you must already have a superhuman machine for that task.

Chasing our own tail with concepts like "reasoning". Let's move the concept a bit - "search". Can LLMs search for novel ideas and discoveries? They do under the right circumstances. You got to provide idea testing environments, the missing ingredient. Search and learn, it's what humans do and AI can do as well.

The whole issue with "reasoning" is that is an incompletely defined concept. Over what domain, what problem space, and what kind of experimental access do we define "reasoning"? Search is better as a concept because it comes packed with all these things, and without conceptual murkiness. Search is scientifically studied to a greater extent.

I don't think we doubt LLMs can learn given training data, we already accuse them of being mere interpolators or parrots. And we can agree to some extent the LLMs can recombine concepts correctly. So they got down the learning part.

And for the searching part, we can probably agree its a matter of access to the search space not AI. It's an environment problem, and even a social one. Search is usually more extended than the lifetime of any agent, so it has to be a cultural process, where language plays a central role.

When you break reasoning/progress/intelligence into "search and learn" it becomes much more tractable and useful. We can also make more grounded predictions on AI, considering the needs for search that are implied, not just the needs for learning.

How much search did AlphaZero need to beat us at go? How much search did humans pack in our 200K years history over 10,000 generations? What was the cost of that journey of search? That kind of questions. In my napkin estimations we solved 1:10000 of the problem by learning, search is 10000x to a million times harder.

Okay, but does that help us reach any meaningful conclusions? For example, okay some AI system doesn't have the capabilities of an auditory cortex or somatosensory cortex. Is there a reason for me to think it needs that?

"Where are we going to find the equivalent data set that will train this other kind of thinking?"

Just a personal opinion, but in my shitty When H.A.R.L.I.E. Was One (and others) unpublished fiction pastiche (ripoff, really), I had the nascent AI stumble upon Cyc as its base for the world and "thinking about how to think."

I never thought that Cyc was enough, but I do think that something Cyc-like is necessary as a component, a seed for growth, until the AI begins to make the transition from the formally defined, vastly interrelated frames and facts in Cyc to being able to growth further and understand the much less formal knowledgebase you might find in, say Wikipedia.

Full agreement with your animal model is only sensible. If you think about macaques, they have a limited range of vocalization once they hit adulthood. Noe that the mothers almost never make a noise at their babies. Lacking language, when a mother wants to train an infant, she hurts it. (Shades of Blindsight there) She picks up the infant, grasps it firmly, and nips at it. The baby tries to get away, but the mother holds it and keeps at it. Their communication is pain. Many animals do this. But they also learn threat displays, the promise of pain, which goes beyond mere carrot and stick.

The more sophisticated multicellular animals (let us say birds, reptiles, mammals) have to learn to model the behavior of other animals in their environment: to prey on them, to avoid being prey. A pond is here. Other animals will also come to drink. I could attack them and eat them. And with the macaques, "I must scare the baby and pain it a bit because I no longer want to breastfeed it."

Somewhere along the line, modeling other animals (in-species or out-species) hits some sort of self-reflection and the recursion begins. That, I think, is a crucial loop to create the kind of intelligence we seek. Here I nod to Egan's Diaspora.

Looping back to your original point about the training data, I don't think that loop is sufficient for an AGI to do anything but think about itself, and that's where something like Cyc would serve as a framework for it to enter into the knowledge that it isn't merely cogito ergo summing in a void, but that it is part of a world with rules stable enough that it might reason, rather than "merely" statistically infer. And as part of the world (or your simulated environment), it can engage in new loops, feedback between its actions and results.

> LLMs absolutely 100% can reason, if we take the dictionary definition; it’s trivial to show their ability to answer non-memorized questions, and the only way to do that is some sort of reasoning.

Um... What? That is a huge leap to make.

'Reasoning' is a specific type of thought process and humans regularly make complicated decisions without doing it. We uses hunches and intuition and gut feelings. We make all kinds of snap assessments that we don't have time to reason through. As such, answering novel questions doesn't necessarily show a system is capable of reasoning.

I see absolutely nothing resumbling an argument for humans having an "ineffable calculator soul", I think that might be you projecting. There is no 'categorical prohibition', only an analysis of the current flaws of specific models.

Personally, my skepticism about imminent AGI has to do believing we may be underestimating the complexity of the software running on our brain. We've reached the point where we can create digital "brains", or atleast portions of them. We may be missing some other pieces of a digital brain, or we may just not have the right software to run on it yet. I suspect it is both but that we'll have fully functional digital brains well before we figure out the software to run on them.

> Even if we accept the most fringe, anthropocentric theories like Penrose & Hammerhoff’s quantum tubules, that’s just a neural network with fancy weights.

First, while it is a fringe idea with little backing it, it's far from the most fringe.

Secondly, it is not at all known that animal brains are accurately modeled as an ANN, any more so than any other Turing-compatible system can be modeled as an ANN. Biological neurons are themselves small computers, like all living cells in general, with not fully understood capabilities. The way biological neurons are connected is far more complex than a weight in an ANN. And I'm not talking about fantasy quantum effects in microtubules, I'm talking about well-established biology, with many kinds of synapses, some of which are "multicast" in a spatially distinct area instead of connected to specific neurons. And about the non-neuronal glands which are known to change neuron behavior and so on.

How critical any of these differences are to cognition is anyone's guess at this time. But dismissing them and reducing the brain to a bigger NN is not wise.

You can't breakdown cognition into just "search" and "learn" without either ridiculously overloading those concepts or leaving a ton out.

> However, it also seems plausible that you could build an abstract representation of the world through studying a vast amount of human language and that'll be a good approximation of the real-world too and furthermore it seems possible that reasoning about that abstract representation can take place in the depths of the layers of a large transformer.

While I agree this is possible, I don't see why you'd think it's likely. I would instead say that I think it's unlikely.

Human communication relies on many assumptions of a shared model of the world that are rarely if ever discussed explicitly, and without which certain concepts or at least phrases become ambiguous or hard to understand.

Sure, when humans use multiple skill to address a specific problem, you can sometimes outperform them by scaling a spefic one of those skills.

When it comes to general intelligence, I think we are trying to run before we can walk. We can't even make a computer with a basic, animal level understanding of the world. Yet we are trying to take a tool that was developed on top of system that already had an understanding of the world and use it to work backwards to give computers an understanding of the world.

I'm pretty skeptical that we're going to succeed at this. I think you have to be able to teach a computer to climb a tree or hunt (subhuman AGI) before you can create superhuman AGI.

Right, but what is also important to remember is while size is important what is also key here is the complexity of a neural circuits. Human brain has a lot more connections and is much more complex.

You seem to be conflating "different hardware" with proof that "language hardware" uses "software" equivalent to LLMs.

LLMs basically become practical when you simply scale compute up, and maybe both regions are "general compute", but language ends up on the "GPU" out of pure necessity.

So to me, these are entirely distinct questions: is the language region able to do general cognitive operations? What happens when you need to spell out "ubiquitous" or declense a foreign word in a language with declension (which you don't have memory patterns for)?

I agree it seems obvious that for better efficiency (size of training data, parameter count, compute ability), human brains use different approach than LLMs today (in a sibling comment, I bring up an example of my kids at 2yo having a better grasp of language rules than ChatGPT with 100x more training data).

But let's dive deeper in understanding what each of these regions can do before we decide to compare to or apply stuff from AI/CS.

It is my understanding that Penrose doesn’t claim that brains are needed for cognition, just that brains are needed for a somewhat nebulous „conscious experience“, which need not have any observable effects. I think that it’s fairly uncontroversial that a machine can produce behavior that is indistinguishable from human intelligence over some finite observation time. The Chinese room speaks Chinese, even if it lacks understanding for some definitions of the term.

>What this tells us for AI is that we need something else besides LLMs.

No this is not true. For two reasons.

1. We call these things LLMs and we train it with language but we can also train it with images.

2. We also know LLMs develop a sort of understanding that goes beyond language EVEN when the medium used for training is exclusively language.

The naming of LLMs is throwing you off. You can call it a Large Language Model but this does not mean that everything about LLMs are exclusively tied only to language.

Additionally we don't even know if the LLM is even remotely similar to the way human brains process language.

No such conclusion can be drawn from this experiment.

> I don't think being tuned to communicate underwater makes them more intelligent than humans.

Your responding to a claim that was never made. The claim was don't assume humans are smarter than whales. Nobody said whales are more intelligent than humans. He just said don't assume.

At times I had impaired brain function (lots of soft neurological issues, finger control, memory loss, balance issues) but surprisingly the core area responsible for mathematical reasoning was spared .. that was a strange sensation, almost schizophrenic.

And yeah it seems that core primitives of intelligence exist very low in our brains. And with people like Michael Levin, there may even be a root beside nervous systems.

This seems quite reasonable, but I recently heard a podcast (https://www.preposterousuniverse.com/podcast/2024/06/24/280-...) that LLMs are more likely to be very good at navigating what they have been trained on, but very poor at abstract reasoning and discovering new areas outside of their training. As a single human, you don't notice, as the training material is greater than everything we could ever learn.

After all, that's what Artificial General Intelligence would at least in part be about: finding and proving new math theorems, creating new poetry, making new scientific discoveries, etc.

There is even a new challenge that's been proposed: https://arcprize.org/blog/launch

> It makes sense that the process of thinking and the process of translating those thoughts into and out of language would be distinct

Yes, indeed. And LLMs seem to be very good at _simulating_ the translation of thought into language. They don't actually do it, at least not like humans do.

>What this tells me is there is clearly no “reasoning” happening whatsoever with either model, despite marketing claiming as such.

Not true. You yourself have failed at reasoning here.

The problem with your logic is that you failed to identify the instances where LLMs have succeeded with reasoning. So if LLMs both fail and succeed it just means that LLMs are capable of reasoning and capable of being utterly wrong.

It's almost cliche at this point. Tons of people see the LLM fail and ignore the successes then they openly claim from a couple anecdotal examples that LLMs can't reason period.

Like how is that even logical? You have contradictory evidence therefore the LLM must be capable of BOTH failing and succeeding in reason. That's the most logical answer.

The connectome and brain mapping efforts might be a better research path for the coming years I guess

I like your premise! And will check out Harlie!

> A pond is here. Other animals will also come to drink. I could attack them and eat them.

Is that the dominant chain, or is the simpler “I’ve seen animals here before that I have eaten” or “I’ve seen animals I have eaten in a place that smelled/looked/sounded/felt like this” sufficient to explain the behavior?

We should look to the animals.

Higher order faculties aside, animals seem like us, just simpler.

The higher functioning ones appear to have this missing thing too. We can see it in action. Perhaps all of them do and it is just harder for us when the animal thinks very differently or maybe does not think as much, feeling more, for example.

----

Now, about that thing... and the controversy:

Given an organism, or machine for this discussion, is of sufficiently robust design and complexity that it can precisely differentiate itself from everything else, it is a being.

This thing we are missing is an emergent property, or artifact that can or maybe always does present when a state of being also presents.

We have not created a machine of this degree yet.

Mother nature has.

The reason for emergence is a being can differentiate sensory input as being from within, such as pain, or touch, and from without, such as light or motion.

Another way to express this is closed loop vs open loop.

A being is a closed loop system. It can experience cause and effect. It can be the cause. It can be the effect.

A lot comes from this closed loop.

There can be the concept of the self and it has real meaning due to the being knowing what is of itself or something, everything else.

This may be what forms consciousness. Consciousness may require a closed loop, and organism of sufficient complexity to be able to perceive itself.

That is the gist of it.

These systems we make are fantastic pieces. They can pattern match and identify relationships between the data given in amazing ways.

But they are open loop. They are not beings. They cannot determine what is part of them, what they even are,or anything really.

I am both consistently amazed and dismayed at what we can get LLM systems to do.

They are tantalizingly close!

We found a piece of how all this works and we are exploiting the cral out of it. Ok fine. Humans are really good at that.

But it will all taper off. There are real limits because we will eventually find the end goal will be to map out the whole problem space.

Who has tried computing that? It is basically all possible human thought. Not going to happen.

More is needed.

And that "more" can arrive at thoughts without having first seen a few bazillion to choose from.

More than a memory.

Needs to be a closed loop, running on its own.

We get its attention, and it responds, or frankly if we did manage any sort of sentience, even a simulation of it, then the fact is it may not respond.

To me, that is the real test.

> What this tells us for AI is that we need something else besides LLMs

I am not convinced it follows. Sure LLMs don’t seem complete however there’s a lot of unspoken inference going on in LLMs that don’t map into a language directly already - the inner layers of the deep neural net that operates on abstract neurons.

Can they reason, or is the volume of training data sufficient for them to match relationships up to appropriate expressions?

Basically, if humans have had meaningful discussions about it, the product of their reasoning is there for the LLM, right?

Seems to me, the "how many R's are there in the word "strawberry" problem is very suggestive of the idea LLM systems cannot reason. If they could, the question is not difficult.

The fact is humans may never have actually discussed that topic in any meaningful way captured in the training data.

And because of that and how specific the question is, the LLM has no clear relationships to map into a response. It just does best case, whatever the math deemed best.

Seems plausible enough to support the opinion LLM'S cannot reason.

What we do know is LLMs can work with anything expressed in terms of relationships between words.

There is a ton of reasoning templates contained in that data.

Put another way:

Maybe LLM systems are poor at deduction, save for examples contained in the data. But there are a ton of examples!

So this is hard to notice.

Maybe LLM systems are fantastic at inference! And so those many examples get mapped to the prompt at hand very well.

And we do notice that and see it like real thinking, not just some horribly complex surface containing a bazillion relationships...

Can't escape the hype.

GP argument seems to be about "thinking" when restricted to knowledge through language, and "possible" is not the same as "likely" or "unlikely" — you are not really disagreeing, since either means "possible".

But conscious experience does produce observable effects.

For that not to be the case, you'd have to take the position that humans experience consciousness and they talk about consciousness but that there is no causal link between the two! It's just a coincidence that the things you find yourself saying about consciousness line up with your internal experience?

https://www.lesswrong.com/posts/fdEWWr8St59bXLbQr/zombies-zo...

Interestingly though for AI, this doesn’t necessarily mean we need a different model architecture. A single large multimodal transformer might be capable of a lot that an LLM is not (besides the multimodality).

I agree we are not limited with the data set size: all humans learn the language with the much smaller language training set (just look at kids and compare them to LLMs).

OTOH, humans (and animals) do get other data feeds (visual, context, touch/pain, smell, internal balance "sensors"...) that we develop as we grow and tie that to learning about language.

Obviously, LLMs won't replicate that since even adults struggle to describe these verbally.

That philosophers talk about p-zombies seems like evidence to me that at least some of them don't believe that consciousness needs to have observable effects that can't be explained without consciousness. I don't say that I believe that too. I don't believe that there is anything particularly special about brains.

Most pre-deep learning architectures had separate modules like "language model", "knowledge base" and "inference component".

Then LLMs came along, and ML folks got rather too excited that they contain implicit knowledge (which, of course, is required to deal with ambiguity). Then the new aspiration as "all in one" and "bigger is better", not analyzing what components are needed and how to orchestrate their interplay.

From an engineering (rather than science) point of view, the "end-to-end black box" approach is perhaps misguided, because the result will be a non-transparent system by definition. Individual sub-models should be connected in a way that retains control (e.g. in dialog agents, SRI's Open Agent Architecture was a random example of such "glue" to tie components together, to name but one).

Regarding the science, I do believe language adds to the power of thinking; while (other) animals can of course solve simple problems without language, language permits us to define layers of abstractions (by defining and sharing new concepts) that goes beyond simple, non-linguistic thoughts. Programming languages (created by us humans somewhat in the image of human language) and the language of mathematics are two examples where we push this even further (beyond the definition of new named concepts, to also define new "DSL" syntax) - but all of these could not come into beying without human language: all formal specs and all axioms are ultimately and can only be formulated in human language. So without language, we would likely be stuck at a very simple point of development, individually and collectively.

EDIT: 2 typos fixed

GP said plausible, which does mean likely. It's possible that there's a teapot in orbit around Jupiter, but it's not plausible. And GP is specifically saying that by studying human language output, you could plausibly learn about the world that have birth to the internal models that language is used to exteriorize.

If we are really nitpicking, they said it's plausible you could build an abstract representation of the world by studying language-based data, but that it's possible it could be made to effectively reason too.

Anyway, it seems to me we are generally all in agreement (in this thread, at least), but are now being really picky about... language :)

Why would he not "assume" that when humans have shaped their world so far beyond what it was, creating intricate layers of art, culture and science; even going into space or in the air? Man collectively tamed nature and the rest of the animal kingdom in a way that no beast ever has.

Anyway, this is just like solipsism, you won't find a sincere one outside the asylum. Every Reddit intellectual writing such tired drivel as "who's to say humans are more intelligent than beasts?" deep down knows the score.

If brain isn't more special than Chinese room, then brain understands Chinese no better than Chinese room?

Is beying another typo?

In my personal learning journey I have been exploring the space of intuitive learning which is dominant in physical skills. Singing requires extremely precise control of actions we can't fully articulate or even rationalise. Teaching those skills requires metaphors and visualising and a whole lot of feedback + trial & error.

I believe that this kind of learning is fundamentally non verbal and we can achieve abstraction of these skills without language. Walking is the most universal of these skills and we learn it before we can speak but if you study it (or better try to program a robot to walk with as many degrees of freedom as the human musculoskeletal system) you will discover that almost all of us don't understand what all the things that go into the "simple" task of walking!

My understanding is that people who are gifted at sports or other physical skills like musical instruments have developed the ability to discover and embed these non verbal abstractions quickly. When I practise the piano and am working on something fast, playing semiquavers at anything above 120bpm is not really conscious anymore in the sense of "press this key then that key"

The concept of arpeggio is verbal but the action is non verbal. In human thought where does verbal and non-verbal start and end? Its probably a continuum

What? I used the term "sensoral" after thinking about what I wanted to communicate. I have no idea if that is a pop psychology term, I didn't google it. I was attempting to communicate that I often think in visual, aural, tactile or olfactory modes, not just visually or via inner monologue, especially when recalling memories.

You're just projecting at this point and stalking previous comments to start arguments. That is exceedingly immature and absolutely against Hacker News guidelines. You need to reevaluate your behavior. Please refrain from continuing to start arguments on previous posts.

There's a lot of other interesting biology besides propagation of electrical signals. Examples include: 1/ Transport of mRNAs (in specialized vesicle structures!) between neurons. 2/ Activation and integration of retrotransposons during brain development (which I have long hypothesized acts as a sort of randomization function for the neural field). 3/ Transport of proteins between and within neurons. This isn't just adventitious movement, either - neurons have a specialized intracellular transport system that allows them to deliver proteins to faraway locations (think >1 meters).

> As a single human, you don't notice, as the training material is greater than everything we could ever learn.

This bias is real. Current gen ai works proportionally well the more known it is. The more training data, the better the performance. When we ask something very specific, we have the impression that it’s niche. But there is tons of training data also on many niche topics, which essentially enhances the magic trick – it looks like sophisticated reasoning. Whenever you truly go “off the beaten path”, you get responses that are (a) nonsensical (illogical) and (b) “pulls” you back towards a “mainstream center point” so to say. Anecdotally of course..

I’ve noticed this with software architecture discussions. I would have some pretty standard thing (like session-based auth) but I have some specific and unusual requirement (like hybrid device- and user identity) and it happily spits out good sounding but nonsensical ideas. Combining and interpolating entirely in the the linguistic domain is clearly powerful, but ultimately not enough.

The “how many R’s are in the word strawberry?” problem can’t be solved by LLMs specifically because they do not have access to the text directly. Before the model sees the user input it’s been tokenized by a preprocessing step. So instead of the string “strawberry”, the model just sees an integer token the word has been mapped to.

Chess is essentially a puzzle. There's a single explicit, quantifiable goal, and a solution either achieves the goal or it doesn't.

Solving puzzles is a specific cognitive task, not a general one.

Language is a continuum, not a puzzle. The problem with LLMs is that testing has been reduced to performance on language puzzles, mostly with hard edges - like bar exams, or letter counting - and they're a small subset of general language use.

Small wonder why you received a sub-optimal response.

Name a creature on earth without one.

Imagine trying to limit, control, or explain a being without familiar cognitive structures.

Is there a reason to care about such unfamiliar modalities of cognition?

> I do believe language adds to the power of thinking; while (other) animals can of course solve simple problems without language, language permits us to define layers of abstractions (by defining and sharing new concepts) that goes beyond simple, non-linguistic thoughts.

Based on my experience with toddlers, a rather smart dog, and my own thought processes, I disagree that language is a fundamental component of abstraction. Of sharing abstractions, sure, but not developing them.

When I'm designing a software system I will have a mental conception of the system as layered abstractions before I have a name for any component. I invent names for these components in order to define them in the code or communicate them to other engineers, but the intuition for the abstraction comes first. This is why "naming things" is one of the hard problems in computer science—because the name comes second as a usually-inadequate attempt to capture the abstraction in language.

The brain is faster than the Chinese room, but other than that, yes, that's the so-called systems reply; Searle's response to it (have the person in the room memorize the instruction book) is beside the point, as you can teach people to perform all sorts of algorithms without them needing to understand the result.

As many people have pointed out, Searle's argument begs the question by tacitly assuming that if anything about the room understands Chinese, it can only be the person within it.

> Why would he not "assume" that when humans have shaped their world so far beyond what it was, creating intricate layers of art, culture and science; even going into space or in the air? Man collectively tamed nature and the rest of the animal kingdom in a way that no beast ever has.

Because whales or dolphins didn’t evolve hands. Hands are a foundational prerequisite for building technology. So if whales or dolphins had hands we don’t know if they would develop technology that can rival us.

Because we don’t know, that’s why he says don’t assume. This isn’t a “deep down we know” thing like your more irrational form of reasoning. It is a logical conclusion: we don’t know. So don’t assume.

I think it’s not entirely accurate to say that we “learn” to walk from a zero state. It’s clear that our DNA has embedded knowledge of how to walk and it develops our body appropriately. Our brains might also have preconditioning to make learning to walk much easier.

Music or sports are more interesting to investigate (in my opinion) since those specific actions won’t be preprogrammed and must be learned independently.

The same way we build abstractions for language in order to perform “telepathy” it seems like for music or sports we build body-specific abstractions. They work similar to words within our own brain but are not something easily communicated since they’re not tied to any language, it’s just a feeling.

I think it’s an interesting point that quite often the best athletes or musicians are terrible coaches. They probably have a much more innate internal language for their body that cannot be communicated easily. Partially, I think, that their body is more different than others which helps them be exceptional. Or that weaker athletes or musicians need to focus much more on lessons from others, so their body language gets tied much closer to human language and that makes it much easier for them to then communicate the lessons they learn to others.

The p-zombie argument is the best-known of a group of conceivability arguments, which ultimately depend on the notion that if a proposition is conceivably true, then there is a metaphysically possible world in which it is true. Skeptics suppose that this is just a complicated way of equivocating over what 'conceivable' means, and even David Chalmers, the philosopher who has done the most to bring the p-zombie argument to wide attention, acknowledges that it depends on the assumption of what he calls 'perfect conceivability', which is tantamount to irrefutable knowledge.

To deal with the awkwardly apparent fact that consciousness certainly seems to have physical effects, zombiephiles challenge the notion that physics is causally closed, so that it is conceivable that something non-physical can cause physical effects. Their approach is to say that the causal closure of physics is not provable, but at this point, the argument has become a lexicographical one, about the definition of the words 'physics' and 'physical' (if one insists that 'physical' does not refer to a causally-closed concept, then we still need a word for the causal closure within which the physical is embedded - but that's just what a lot of people take 'physical' to mean in the first place.) None of the anti-physicalists have been able, so far, to shed any light on how the mind is causally effective in the physical world.

You might be interested in the late Daniel Dennett's "The Unimagined Preposterousness of Zombies": https://dl.tufts.edu/concern/pdfs/6m312182x

It is very naïve to think that the availability of such tools isn't partly responsible for that intelligence; “We shape our tools and thereafter our tools shape us”. And it seems too man-centric of an excuse: you can see all our civilization being built on hands so you state that there can't be a way without.

The "they MIGHT be as intelligent, just lacking hands" theory can't have the same weight as "nah" in an honest mind seeing the overwhelming clues (yes, not proof, if that's what you want) against it. Again, same way that you can't disprove solipsism.

All well said, and I agree on many of your final points! But you beautifully highlighted my issue at the top:

  'Reasoning' is a specific type of thought process 

If so, what exactly is it? I don’t need a universally justified definition, I’m just looking for an objective, scientific one. A definition that would help us say for sure that a particular cognition is or isn’t a product of reason.

I personally have lots of thoughts on the topic and look to Kant and Hegel for their definitions of reason as the final faculty of human cognition (after sensibility, understanding, and judgement), and I even think there’s good reason (heh) to think that LLMs are not a great tool for that on their own. But my point is that none of the LLM critics have a definition anywhere close to that level of specificity.

Usually, “reason” is used to mean “good cognition”, so “LLMs can’t reason” is just a variety of cope/setting up new goalposts. We all know LLMs aren’t flawless or infinite in their capabilities, but I just don’t find this kind of critique specific enough to have any sort of scientific validity. IMHO

Well, it’s the basis of programming languages. That seems pretty helpful :) Otherwise it’s hard to measure what exactly “real world utility” looks like. What have the other branches of linguistics brought us? What has any human science brought us, really? Even the most empirical one, behavioral psychology, seems hard to correlate with concrete benefits. I guess the best case would be “helps us analyze psychiatric drug efficacy”?

Generally, I absolutely agree that he is not humble in the sense of expressing doubt about his strongly held beliefs. He’s been saying pretty much the same things for decades, and does not give much room for disagreement (and ofc this is all ratcheted up in intensity in his political stances). I’m using humble in a slightly different way, tho: he insists on qualifying basically all of his statements about archaeological anthropology with “we don’t have proof yet” and “this seems likely”, because of his fundamental belief that we’re in a “pre-Galilean” (read: shitty) era of cognitive science.

In other words: he’s absolutely arrogant about his core structural findings and the utility of his program, but he’s humble about the final application of those findings to humanity.

Not sure about that. The same abstract model could be used for both (symbols generated in sequence). For language the symbols have meaning in the context of language. For non-language thought they don't. Nature seems to work this way in general: re-using/purposing the same underlying mechanism over and over at different levels in the stack. All of this could be a fancy version of very old hardware that had the purpose of controlling swimming direction in fish. Each symbol is a flick of the tail.

The difference is that my conclusion is logical and yours is an assumption.

I like to think of the non-verbal portions as the biological equivalents of ASICs. even skills like riding a bicycle might start out as conscious effort (a vision model, a verbal intention to ride and a reinforcement learning teacher) but is then replaced by a trained model to do the job without needing the careful intentional planning. some of the skills in the bag of tricks are fine tuned by evolution.

ultimately, there's no reason that a general algorithm couldn't do the job of a specific one, just less efficiently.

You highlight an expectation that the “truer intelligence” is a singular device, once isolated would mobilize ultimate AGI.

All intelligence is the mitigation of uncertainty (the potential distributed problem.) if it does not mitigate uncertainty it is not intelligence, it is something else.

Intelligence is a technology. For all life intelligence and the infrastructure of performing work efficiently (that whole entropy thing again) is a technology. Life is an arms race to maintain continuity (identity, and the very capacity of existential being.)

The modern problem is achieving reliable behavioral intelligence (constrained to a specific problem domain.) AGI is a phantasm. What manifestation of intelligence appears whole and complete and is always right? These are the sorts of lies you tell yourself, the ones that get you into trouble. They distract from tangible real world problems, perhaps causing some of them. True intelligence is a well calibrated “scalar” domain specific problem (uncertainty) reducer. There are few pressing idempotent obstructions in the real world.

Intelligence is the mitigation of uncertainty.

Uncertainty is the domain of negative potential (what,where,why,how?)

Mitigation is the determinant resolve of any constructive or destructive interference affecting (terminal resolve within) the problem domain.

Examples of this may be piled together mountains high, and you may call that functional AGI, though you would be self deceiving. At some point “good enough” may be declared for anything so passing as yourselves.

I feel you are putting too much emphasis on the importance and primacy of having a definition of words like 'reasoning'.

As humanity has struggled to understand the world, it has frequently given names to concepts that seem to matter, well before it is capable of explaining with any sort of precision what these things are, and what makes them matter - take the word 'energy', for example.

It seems clear to me that one must have these vague concepts before one can begin to to understand them, and also that it would be bizarre not to give them a name at that point - and so, at that point, we have a word without a locked-down definition. To insist that we should have the definition locked down before we begin to investigate the phenomenon or concept is precisely the wrong way to go about understanding it: we refine and rewrite the definitions as a consequence of what our investigations have discovered. Again, 'energy' provides a useful case study for how this happens.

A third point about the word 'energy' is that it has become well-defined within physics, and yet retains much of its original vagueness in everyday usage, where, in addition, it is often used metaphorically. This is not a problem, except when someone makes the lexicographical fallacy of thinking that one can freely substitute the physics definition into everyday speech (or vice-versa) without changing the meaning.

With many concepts about the mental, including 'reasoning', we are still in the learning-and-writing-the-definition stage. For example, let's take the definition you bring up: reasoning as good cognition. This just moves us on to the questions of what 'cognition' means, and what distinguishes good cognition from bad cognition (for example, is a valid logical argument predicated on what turns out to be a false assumption an example of reasoning-as-good-cognition?) We are not going to settle the matter by leafing through a dictionary, any more than Pedro Carolino could write a phrase book just from a Portugese-English dictionary (and you are probably aware that looking up definitions-of-definitions recursively in a dictionary often ends up in a loop.)

A lot of people want to jump the gun on this, and say definitively either that LLMs have achieved reasoning (or general intelligence or a theory of mind or even consciousness, for that matter) or that they have not (or cannot.) What we should be doing, IMHO, is to put aside these questions until we have learned enough to say more precisely what these terms denote, by studying humans, other animals, and what I consider to be the surprising effectiveness of LLMs - and that is what the interviewee in the article we are nominally discussing here is doing.

You entered this thread by saying (about the paper underlying an article in Ars Tech [1]) I’ll pop in with a friendly “that research is definitely wrong”. If they want to prove that LLMs can’t reason..., but I do not think there is anything like that claim in the paper itself (one should not simply trust what some person on HN says about a paper. That, of course, goes as much for what I say about it as what the original poster said.) To me, this looks like the sort of careful, specific and objective work that will lead to us a better understanding of our concepts of the mental.

[1] https://arxiv.org/pdf/2410.05229

Dolphins, Orcas, whales and other intelligent cetaceans do not have Hands and live in an environment without access to a technological accelerator like fire.

The absence of both of these things is an incredible crippler for technological development. It doesn't matter how intelligent you are, you're never going to achieve much technologically without these.

I don't think brain size correlations is as straightforward as 'bigger = better' every time but we simply don't know how intelligent most of these species are. Land and Water are completely different beasts.

> This has been suspected for years, but now there's an experimental result.

You would think the whole "split-brain" thing would have been the first clue; apparently not.

You are getting derailed because of the name we've chosen to call these models but only the first few and last few layers of LLMs deal with tokens. The rest deal with abstract representations and models learnt during training. Language goes in and Language comes out but Language is not the in-between for either LLMs or Humans.

> So if someone figures out to do this, it will probably take less hardware than an LLM.

We have, it's called DreamCoder. There's a paper and everything.

Everything needed for AGI exists today, people simply have (incorrect) legacy beliefs about cognition that are holding them back (e.g. "humans are rational").

https://arxiv.org/abs/2006.08381

> No idea how to do this

We need to add the 5 senses, of which we have now image, audio and video understanding in LLMs. And for agentic behavior they need environments and social exposure.

Intelligence isn't measured by ability to create technology or use tools.

Intelligence is the ability to use experience to predict your environment and the outcomes of your own actions. It's a tool for survival.

> the "end to end black box" approach is perhaps misguided, because the result will be a non transparent system by definition

A black box that works in human language and can be investigated with perturbations, embedding visualizations and probes. It explains itself as much ore more than we can.

Okay and how have we determined we have more intelligence than those species with this measure ?

Clearly we haven't, given that there is very little agreement as to what intelligence is. This is just my definition, although there's a lot behind why I define it this way.

However, I do think that a meaningful intelligence comparison between humans and dolphins, etc, would conclude that we are more intelligent, especially based on our reasoning/planning (= multi-step prediction) abilities, which allows us not only to predict our environment but also to modify it to our desires in very complex ways.

The conception here is that one's layered abstractions is basically an informal mathematics... which is formally structured... which is a formal grammar. It's your internal language, using internal symbols instead of English names.

Remember in CS theory, a language is just a set of strings. If you think in pictures that is STILL a language if your pictures are structured.

So I'm really handwaving the above just to suggest that it all depends on the assumptions that each expert is making in elucidating this debate which has a long history.

>However, I do think that a meaningful intelligence comparison between humans and dolphins, etc, would conclude that we are more intelligent, especially based on our reasoning/planning (= multi-step prediction) abilities

I'm not sure how you would make meaningful comparisons here. We can't communicate to them as they communicate and we live in almost completely different environments. Any such comparison would be extremely biased to us.

>which allows us not only to predict our environment but also to modify it to our desires in very complex ways.

We modify our environment mostly through technology. Intelligence is a big part of technology sure but it's not the only part of it and without the other parts (hands with opposable thumbs, fire etc), technology as we know it wouldn't exist and our ability to modify the environment would seem crippled to any outside observer regardless of how intelligent we may be.

It's not enough to think that the earth revolves around the sun, we need to build the telescopes (with hands and materials melted down and forged with fire) to confirm it.

It's not enough to dream and devise of flight, we need the fire to create the materials that we dug with our hands and the hands to build them.

It's not enough to think that Oral communication is insufficient for transmitting information through generations. What else will you do without opposable thumbs or an equivalent ?

Fire is so important for so many reasons but one of the biggest is that it was an easy source of large amounts of energy that allowed us to bootstrap technology. Where's that easy source of energy underwater ?

Without all the other aspects necessary for technology, we are relegated to hunter/gatherer levels of influencing the environment at best. Even then, we still crafted tools that creatures without opposable thumbs would never be able to craft.

> What this tells us for AI is that we need something else besides LLMs.

Despite being an LLM skeptic of sorts, I don’t think that necessarily follows. The LLM matrix multiplication machinery may well be implementing an equivalent of the human non-language cognitive processing as a side effect of the training. Meaning, what is separated in the human brain may be mixed together in an LLM.

> don’t need a universally justified definition, I’m just looking for an objective, scientific one. A definition that would help us say for sure that a particular cognition is or isn’t a product of reason.

Unfortunately, you won't get one. We simply don't know enough about cognition to create rigourous definitions of the type you are looking for.

Instead, this paper, and the community in general are trying to perform practical capability assessments. The claim that the GSM8k measures "mathematical reasoning" or "logical reasoning" didn't come from the skeptics.

Alan Turring didn't try to define intelligence, he created a practical test that he thought would be a good benchmark. These days we believe we have better ones.

> I just don’t find this kind of critique specific enough to have any sort of scientific validity. IMHO

"Good cognition" seems like dismisal of a definition, but this is exactly the definition that the people working on this care about. They are not philosphers, they are engineers who are trying to make a system "better" so "good cognition" is exactly what they want.

The paper digs into finding out more about what types of changes impacts peformance on established metrics. The "noop" result is pretty interesting since "relevancy detection" isn't something we commonly think of as key to "good cognition", but a consequence of it.

I'm curious why "simulation" isn't the extra thing needed? Yes, we need language to communicate ideas. But you can simulate in your mind things happening that you don't necessarily have words for, yet. Right?

I don't think motor skills are a good object to use in an argument about verbal vs non-verbal thinking. We have large regions of our brains primarily dedicated to motor skills and you can't argue that humans are any more talented or capable at controlling our bodies than other animals, we're actually rather poor performers in this area. You're right to say that you aren't conscious of the very highly trained movements you are making because they likely have only a tenuous connection with any part of your brain that we would recognize as possessing consciousness or thought, they are mostly learned reflexes and responses to internal and external stimuli at this point like a professional baseball player who can automatically catch a ball flying at him before he's even aware of it.

> conception here is that one's layered abstractions is basically an informal mathematics... which is formally structured... which is a formal grammar. It's your internal language, using internal symbols instead of English names

Unless we're getting metaphysical to the point of describing quantum systems as possessig a language, there are various continuous analog systems that can compute without a formal grammar. The language system could be the one that thinks in discrete 'tokens'; the conscious system something more complex.

Another angle to look at intelligence is that not all species need it, or need it to same degree. If you are a cow, or a crocodile, then you are a 1-trick grass-munching or zebra-munching pony, and have no need for intelligence. A generalist species like humans, that lives in a hugely diverse set of environments, with a hugely diverse set of food sources, has evolved intelligence (which in turn supports further generalization) to cope with this variety.

At least to our own perception, and degree of understanding, it would appear that the ocean habitat(s) of dolphins are far less diverse and demanding. Evidentially complex enough to drive their intelligence though, so perhaps we just don't understand the complexity of what they've evolved to do.

Success doesn’t imply that “reasoning” was involved, and the definition of reasoning is extremely important.

Apple’s recent research summarized here [0] is worth a read. In short, they argue that what LLMs are doing is more akin to advanced pattern recognition than reasoning in the way we typically understand reasoning.

By way of analogy, memorizing mathematical facts and then correctly recalling these facts does not imply that the person actually understands how to arrive at the answer. This is why “show your work” is a critical aspect of proving competence in an education environment.

An LLM providing useful/correct results only proves that it’s good at surfacing relevant information based on a given prompt. That fact that it’s trivial to cause bad results by making minor but irrelevant changes to a prompt points to something other than a truly reasoned response, i.e. a reasoning machine would not get tripped up so easily.

- [0] https://x.com/MFarajtabar/status/1844456880971858028

Transformers are just sequence predictors, it doesn’t need to be language, increasingly it’s not

It’s AGI via transformer

> What this tells us for AI is that we need something else besides LLMs.

An easy conclusion to jump to but I believe we need to be more careful. Nothing in these findings proves conclusively that non-verbal reasoning mechanism equivalent to humans couldn't evolve in some part of a sufficiently large ANN trained on text and math. Even though verbal and non-verbal reasoning occurs in two distinct parts of the brain, it doesn't mean they're not related.

You’re still suffering from the biases of the parent poster. You are picking and choosing papers that illustrate failure instances when there are also an equal amount of papers that verify successful instances.

It’s bloody obvious that when I classify success I mean that the llm is delivering a correct and unique answer for a novel prompt that doesn’t exist in the original training set. No need to go over the same tired analogies that have been regurgitated over and over again that you believe LLMs are reusing memorized answers. It’s a stale point of view. The overall argument has progressed further then that and we now need more complicated analysis of what’s going on with LLMs

Sources: https://typeset.io/papers/llmsense-harnessing-llms-for-high-...

https://typeset.io/papers/call-me-when-necessary-llms-can-ef...

And these two are just from a random google search.

I can find dozens and dozens of papers illustrating failures and successes of LLMs which further nails my original point. LLMs both succeed and fail at reasoning.

The main problem right now is that we don’t really understand how LLMs work internally. Everyone who claims they know LLMs can’t reason are just making huge leaps of irrational conclusions because not only does their conclusion contradict actual evidence but they don’t even know how LLMs work because nobody knows.

We only know how LLMs work at a high level and we only understand these things via the analogy of a best fit curve in a series of data points. Below this abstraction we don’t understand what’s going on.

> Name a creature on earth without one.

Anything that doesn't have a spine, I'm pretty sure.

Also if we look at just auditory, tons of creatures are deaf and don't need that.

> Imagine trying to limit, control, or explain a being without familiar cognitive structures.

I don't see why any of that that affects whether it's intelligent.

What part of AI today leads you to believe that an AGI would be capable of self directed creativity? Today that is impossible - no AI is truly generating "new" stuff, no poetry is constructed creatively, no images are born from a feeling, inspiration is only part of AI generation is you consider it utilizing it's training data, which isn't actually creativity.

I'm not sure why everyone assumes an AGI would just automatically do creativity considering most people are not very creative, despite them quite literally being capable, most people can't create anything. Why wouldn't an AGI have the same issues with being "awake" that we do? Being capable of knowing stuff - as you pointed out, far more facts than a person ever could, I think an awake AGI may even have more "issues" with the human condition than us.

Also - say an AGI comes into existence that is awake, happy and capable of truly original creativity - why tf does it write us poetry? Why solve world hunger - it doesn't hunger. Why cure cancer - what can cancer do to to it?

AGI as currently envisioned is a mythos of fantasy and science fiction.

Like what is magic - it turns out to be the ability to go from interior thoughts to stuff happening in the shared world - physics is just the mechanism of the particular magical system we have.

Language models would seem to be exquisitely tied to the way that evolved intelligence has formulated its society and training.

An Ab Initio AGI would maybe be free of our legacy, but LLMs certainly are not.

I would expect a ship-like intelligence a la the Culture novels to have non-English based cognition. As far as we can tell, our own language generation is post-hoc explanation for thought more so than the embodiment of thought.

This is one of my favorite comments I've ever read on HN.

The first three paragraphs you wrote very succinctly and obviously summarize the fundamental flaw of our modern science - that it can't make leaps, at all.

There is no leap of faith in science but there is science that requires such leaps.

We are stuck bc those most capable of comprehending concepts they don't understand and are unexplainable - they won't allow themselves to even develop a vague understanding of such concepts. The scientific method is their trusty hammer and their faith in it renders all that isn't a nail unscientific.

Admitting that they don't kno enough would be akin to societal suicide of their current position - the deciders of what is or isn't true, so I don't expect them to withhold their conclusions til they are more able to.

They are the "priest class" now ;)

I agree with your humble opinion - there is much more we could learn if that was our intent and considering the potential of this, I think we absolutely ought to make certain that we do everything in our power to attain the best possible outcomes of these current and future developments.

Transparent and honest collaboration for the betterment of humanity is the only right path to an AGI god - to oversimplify a lil bit.

Very astute, well formulated position, presented in accessible language and with humility even!

Well done.

This is actually exactly what is needed. We think the dataset is the primary limitation to an LLMs capability but in reality we are only developing one part of their "intelligence" - a functional and massive model isn't the end of their training - its kinda just the beginning.

Could be! But then there are ambushes, driving prey into the claws of hidden allies, and so forth. Modeling the behavior of other animals will have to occur without place for many instances.

Claim is LLM exhibit reasoning, particularly in coding and logic. Observation is mere parroting of training data. Observations trump claims.

I’ll say the unholy combination of managing the python GIL, concurrency, and connection reuse is not my favorite topic.

Evolution is a blind, dumb optimizer. You can have a mutation that is over-kill and if it doesn't actively impede you in some way, it just stays. It's not like it goes, "Ok we need to reduce this to the point where it's just beneficial enough etc".

That said, i definitely would not say the Ocean is particularly less diverse or demanding.

Even with our limited understanding, there must be adaptations for Pressure, Salinity, light, Energy, Buoyancy, Underwater Current etc that all vary significantly by depth and location.

And the bottlenose dolphin for instance lives in every ocean of the world except the Arctic and the Antarctic oceans.

I mean, the QKV part of transformers is like an "ASIC" ... well, for an (approximate) lookup table.

(also important to note that NNs/LLMs operate on... abstract vectors, not "language" -- not relevant as a response to your post though).

Conclusion noted: nuke the whales before they nuke us.

(/s)

Read the parent post. The claim is LLMs can't reason.

The evidence is using one instance of the LLM parroting training data while completely ignoring contradicting evidence where the LLM created novel answers to novel prompts out of thin air.

>Observations trump claims.

No. The same irrational hallucinations that plague LLMs are plaguing human reasoning and trumping rational thinking.

> You can have a mutation that is over-kill and if it doesn't actively impede you in some way, it just stays.

Right, but big brains do actively impede you - they require a lot of energy, so there needs to be some offsetting benefit.

Must be my lying’ eyes, fooling me once again.

I think my point stands, despite a poor example.[0]

Other examples exist.

[0]That example is due to tokenization. DoH! I knew better too.

Ah well.

Agreed: Perhaps we aught to be studying cognition of creatures without spines before we claim to replicate or understand cognition of creatures with them.

Presumably they have some sort biological input processing or sensory inputs. They don't eat data.

That's based on a well known fallacy, because analog models cannot exceed the computational power of Turing machines. The alternative position is Penrose who thinks quantum tubules are responsible for consciousness and thus somehow more powerful than TMs.

> The main problem right now is that we don’t really understand how LLMs work internally.

Right, and this is why claims that models are “reasoning” can’t be taken at face value. This space is filled with overloaded terms and anthropomorphic language that describes some behavior of the LLM but this doesn’t justify a leap to the belief that these terms actually represent the underlying functionality of the model, e.g. when terms like “hallucinate”, “understand”, etc. are used, they do not represent the biological processes these ideas stem from or carry the implications of a system that mimics those processes.

> Everyone who claims they know LLMs can’t reason are just making huge leaps of irrational conclusions because not only does their conclusion contradict actual evidence but they don’t even know how LLMs work because nobody knows.

If you believe this to be true, you must then also accept that it’s equally irrational to claim these models are actually “reasoning”. The point of citing the Apple paper was that there’s currently a lack of consensus and in some cases major disagreement about what is actually occurring behind the scenes.

Everything you’ve written to justify the idea that reasoning is occurring can be used against the idea that reasoning is occurring. This will continue to be true until we gain a better understanding of how these models work.

The reason the Apple paper is interesting is because it’s some of the latest writing on this subject, and points at inconvenient truths about the operation of these models that at the very least would indicate that if reasoning is occurring, it’s extremely inconsistent and unreliable.

No need to be combative here - aside from being against HN guidelines, there just isn’t enough understanding yet for anyone to be making absolute claims, and the point of my comment was to add counterpoints to a conversation, not make some claim about the absolute nature of things.

actually I think you are on to something - abstract vectors are the tokens of thought - mentalese if you've read any Dennett.

>If you believe this to be true, you must then also accept that it’s equally irrational to claim these models are actually “reasoning”.

If a novel low probability conclusion that is correct was arrived at from a novel prompt where neither the prompt nor the conclusion existed in the training set, THEN by logic the ONLY possible way the conclusion was derived was through reasoning. We know this, but we don't know HOW the model is reasoning.

The only other possible way that an LLM can arrive at low probability conclusions is via random chance.

>The point of citing the Apple paper was that there’s currently a lack of consensus and in some cases major disagreement about what is actually occurring behind the scenes.

This isn't true. I quote the parent comment:

   "What this tells me is there is clearly no “reasoning” happening whatsoever with either model, despite marketing claiming as such." 

Parent is clearly saying LLMs can't reason period.

>Everything you’ve written to justify the idea that reasoning is occurring can be used against the idea that reasoning is occurring. This will continue to be true until we gain a better understanding of how these models work.

Right and I took BOTH pieces of contradictory evidence into account and I ended up with the most logical conclusion. I quote myself:

   "You have contradictory evidence therefore the LLM must be capable of BOTH failing and succeeding in reason. That's the most logical answer."

>The reason the Apple paper is interesting is because it’s some of the latest writing on this subject, and points at inconvenient truths about the operation of these models that at the very least would indicate that if reasoning is occurring, it’s extremely inconsistent and unreliable.

Right. And this, again, was my conclusion. But I took it a bit further. Read again what I said in the first paragraph of this very response.

>No need to be combative here - aside from being against HN guidelines, there just isn’t enough understanding yet for anyone to be making absolute claims, and the point of my comment was to add counterpoints to a conversation, not make some claim about the absolute nature of things.

You're not combative and neither am I. I respect your analysis here even though you dismissed a lot of what I said (see quotations) and even though I completely disagree and I believe you are wrong.

I think there's a further logical argument you're not realizing and I pointed it out in the first paragraph. LLMs are arriving at novel answers from novel prompts that don't exist in the data set. These novel answers have such low probability of existing via random chance that the ONLY other explanation for it is covered by the broadly defined word: "reasoning".

Again, there is also evidence of prompts that aren't arrived at via reasoning, but that doesn't negate the existence of answers to prompts that can only be arrived via reasoning.

It's a fair point that Chomsky's ideas about grammars are used in parsing programming languages. But linguistics is supposed to deal with natural languages -- what has Chomskyan linguistics accomplished there?

Contrast to the statistical approach. It's easy to point to something like Google translate. If Chomsky's approach gave us a tool like that, I'd have no complaint. But my sense is that it just hasn't panned out.

"Language models can explain neurons in language models" https://news.ycombinator.com/item?id=35877402#35886145 :

> Recent work has revealed that the neural activity patterns correlated with sensation, cognition, and action often are not stable and instead undergo large scale changes over days and weeks—a phenomenon called representational drift.

[...]

So, I'm not sure how conclusive this fmri activation study is either.

Though, is there a proto language that's not even necessary for the given measured aspects of condition?

Which artificial network architecture best approximates which functionally specialized biological neutral networks?

OpenCogPrime:KnowledgeRepresentation > Four Types of Knowledge: https://wiki.opencog.org/w/OpenCogPrime:KnowledgeRepresentat... :

> Sensory, Procedural, Episodic, Declarative

From https://news.ycombinator.com/item?id=40105068#40107537 re: cognitive hierarchy and specialization :

> But FWIU none of these models of cognitive hierarchy or instruction are informed by newer developments in topological study of neural connectivity;

> analog models cannot exceed the computational power of Turing machines

There is no reason to assume consciousness is Turing computable [1].

[1] https://en.m.wikipedia.org/wiki/Church%E2%80%93Turing_thesis