Just read Google’s Nested Learning paper and their Continuum Memory System (CMS) concept is next level!!
Instead of the binary short-term/long-term memory we typically build, CMS uses a spectrum of modules that update at different frequencies.
Fast-updating for immediate context, slow-consolidating for longer-term knowledge. Their proof-of-concept (Hope framework) shows agents that continuously learn without catastrophic forgetting.
This is the architecture that’s coming. It’s gonna take some time to get here, but worth keeping in mind as we build now and as we watch future model releases.