Titans: Google Creates Long Term Memory for Use With Transformers

"""

The paper presents a new neural long-term memory module that learns to memorize historical context and helps an attention mechanism attend to the current context while utilizing long past information. The authors argue that attention, due to its limited context but accurate dependency modeling, performs as a short-term memory, while the neural memory, due to its ability to memorize data, acts as a long-term, more persistent memory. Based on these two modules, the authors introduce a new family of architectures called Titans, which effectively incorporate memory into the architecture. The experimental results show that Titans outperform Transformers and recent modern linear recurrent models on various tasks, including language modeling, common-sense reasoning, genomics, and time series forecasting.

"""

https://arxiv.org/abs/2501.00663

0 comments