←back to thread

Google Titans architecture, helping AI have long-term memory

(research.google)
584 points Alifatisk | 5 comments | 07 Dec 25 12:23 UTC | HN request time: 0.001s | source
Show context
okdood64 ◴[07 Dec 25 14:05 UTC] No.46181759[source]▶
From the blog:

https://arxiv.org/abs/2501.00663

https://arxiv.org/pdf/2504.13173

Is there any other company that's openly publishing their research on AI at this level? Google should get a lot of credit for this.

replies(12): >>46181829 #>>46182057 #>>46182168 #>>46182358 #>>46182633 #>>46183087 #>>46183462 #>>46183546 #>>46183827 #>>46184875 #>>46186114 #>>46189989 #
mapmeld ◴[07 Dec 25 15:02 UTC] No.46182168[source]▶
Well it's cool that they released a paper, but at this point it's been 11 months and you can't download a Titans-architecture model code or weights anywhere. That would put a lot of companies up ahead of them (Meta's Llama, Qwen, DeepSeek). Closest you can get is an unofficial implementation of the paper https://github.com/lucidrains/titans-pytorch
replies(7): >>46182351 #>>46182946 #>>46184154 #>>46185017 #>>46186942 #>>46187280 #>>46188385 #
alyxya ◴[07 Dec 25 16:34 UTC] No.46182946[source]▶
The hardest part about making a new architecture is that even if it is just better than transformers in every way, it’s very difficult to both prove a significant improvement at scale and gain traction. Until google puts in a lot of resources into training a scaled up version of this architecture, I believe there’s plenty of low hanging fruit with improving existing architectures such that it’ll always take the back seat.
replies(5): >>46183227 #>>46184404 #>>46184696 #>>46186138 #>>46186853 #
1. tyre ◴[07 Dec 25 20:11 UTC] No.46184696[source]▶
Google is large enough, well-funded enough, and the opportunity is great enough to run experiments.

You don't necessarily have to prove it out on large foundation models first. Can it beat out a 32b parameter model, for example?

replies(1): >>46185008 #
2. swatcoder ◴[07 Dec 25 20:48 UTC] No.46185008[source]▶
Do you think there might be an approval process to navigate when experiments costs might run seven or eight digits and months of reserved resources?

While they do have lots of money and many people, they don't have infinite money and specifically only have so much hot infrastructure to spread around. You'd expect they have to gradually build up the case that a large scale experiment is likely enough to yield a big enough advantage over what's already claiming those resources.

replies(2): >>46189610 #>>46191181 #
3. dpe82 ◴[08 Dec 25 08:01 UTC] No.46189610[source]▶
I would imagine they do not want their researchers unnecessarily wasting time fighting for resources - within reason. And at Google, "within reason" can be pretty big.
replies(1): >>46190731 #
4. howdareme ◴[08 Dec 25 10:34 UTC] No.46190731{3}[source]▶
I mean looking antigravity, jules & gemini cli, they have have no problem with their developers fighting for resources
5. nl ◴[08 Dec 25 11:45 UTC] No.46191181[source]▶
I mean you'd think so, but...

> In fact, the UL2 20B model (at Google) was trained by leaving the job running accidentally for a month.

https://www.yitay.net/blog/training-great-llms-entirely-from...