For the amount of theoretical work behind those Mamba2 blocks (I can barely understand their paper on the subject), those are some extremely modest performance gains.
Attention remains king.
replies(2):
Attention remains king.
Yannic Kilcher has a new video touching on Mamba in an intuitive way.