We should look to the animals.
Higher order faculties aside, animals seem like us, just simpler.
The higher functioning ones appear to have this missing thing too. We can see it in action. Perhaps all of them do and it is just harder for us when the animal thinks very differently or maybe does not think as much, feeling more, for example.
----
Now, about that thing... and the controversy:
Given an organism, or machine for this discussion, is of sufficiently robust design and complexity that it can precisely differentiate itself from everything else, it is a being.
This thing we are missing is an emergent property, or artifact that can or maybe always does present when a state of being also presents.
We have not created a machine of this degree yet.
Mother nature has.
The reason for emergence is a being can differentiate sensory input as being from within, such as pain, or touch, and from without, such as light or motion.
Another way to express this is closed loop vs open loop.
A being is a closed loop system. It can experience cause and effect. It can be the cause. It can be the effect.
A lot comes from this closed loop.
There can be the concept of the self and it has real meaning due to the being knowing what is of itself or something, everything else.
This may be what forms consciousness. Consciousness may require a closed loop, and organism of sufficient complexity to be able to perceive itself.
That is the gist of it.
These systems we make are fantastic pieces. They can pattern match and identify relationships between the data given in amazing ways.
But they are open loop. They are not beings. They cannot determine what is part of them, what they even are,or anything really.
I am both consistently amazed and dismayed at what we can get LLM systems to do.
They are tantalizingly close!
We found a piece of how all this works and we are exploiting the cral out of it. Ok fine. Humans are really good at that.
But it will all taper off. There are real limits because we will eventually find the end goal will be to map out the whole problem space.
Who has tried computing that? It is basically all possible human thought. Not going to happen.
More is needed.
And that "more" can arrive at thoughts without having first seen a few bazillion to choose from.