Google Titans architecture, helping AI have long-term memory

See also Hope:

In the previous sections, we first discussed Continuum Memory System (CMS) that allows for more persistent storage of memories and defines memory as a spectrum of blocks with different frequencies of update. Due to the larger capacity and constraints for scaling the parameters, often CMS requires simple learning rule but higher capacity to store more persistent knowledge. On the other hand, in the previous section, we discussed the design of a self-modifying Titans, where it can generate its own keys and so learning update to better adapt to the context. Contrary to CMS, the self-modifying Titans has a small capacity but is using a complex and expressive learning rule. Accordingly, these two systems seem to be complementary and their combination can enhance the model expressiveness from different aspects.

To this end, we present Hope architecture: A neural learning module that incorporates self-modifying Titans followed by Continuum Memory System.

https://research.google/blog/introducing-nested-learning-a-n...

For most papers, the main idea can be described in 1-2 sentences, sort of "we did X using Y".

That doesn't work for HOPE - a short summary can't explain what it actually does besides "self-modifying" and "continuum memory".

So it seems to be an innovation of Transformers calibre, really big (if true). It's definitely not "transformer but with such-and-such modification".

Gemini came up with a following visual metaphor for the difference:

> Transformer is a series of frozen glass panes (the weights) and a scratchpad (the attention) where it writes notes about the current text.

> The HOPE architecture involves no scratchpad. Instead, the glass panes themselves are made of smart liquid. As the data flows through, the first pane reshapes itself instantly. The second pane reshapes itself slowly. And the mechanism deciding how to reshape them is itself a tiny, intelligent machine, not just a basic math rule.