Problems the AI industry is not addressing adequately

Also, AGI is not just around the corner. We need artificial comprehension for that, and we don't even have a theory how comprehension works. Comprehension is the fusing of separate elements into new functional wholes, dynamically abstracting observations, evaluating them for plausibility, and reconstituting the whole - and all instantaneously, for security purposes, of every sense constantly. We have no technology that approaches that.

We only have two computational tools to work with - deterministic and random behavior. So whatever comprehension/understanding/original thought/consciousness is, it's some algorithmic combination of deterministic and random inputs/outputs.

I know that sounds broad or obvious, but people seem to easily and unknowingly wander into "Human intelligence is magically transcendent".

What you state is called the Physical Church-Turing Thesis, and it's neither obvious nor necessarily true.

I don't know if you're making it, but the simplest mistake would be to think that you can prove that a computer can evaluate any mathematical function. If that were the case then "it's got to be doable with algorithms" would have a fairly strong basis. Anything the mind does that an algorithm can't would have to be so "magically transcendent" that it's beyond the scope of the mathematical concept of "function". However, this isn't the case. There are many mathematical functions that are proven to be impossible for any algorithm to implement. Look up uncomputable functions you're unfamiliar with this.

The second mistake would be to think that we have some proof that all physically realisable functions are computable by an algorithm. That's the Physical Church-Turing Thesis mentioned above, and as the name indicates it's a thesis, not a theorem. It is a statement about physical reality, so it could only ever be empirically supported, not some absolute mathematical truth.

It's a fascinating rabbit hole if you're interested - what we actually do and do not know for sure about the generality of algorithms.