Sleep framework: A new way for LLMs to retain knowledge in continual learning

What happened

Google Research announced a new way for LLMs to learn over time. Most LLMs are fixed after training, so they stop learning new facts. The team built a sleep phase: the model alternates between an active (wake) phase, where it takes in and handles inputs, and a sleep phase, where it pauses new input and moves short-term information into long-term memory.

The problem being solved

Today's LLMs learn in two main ways. Pre-training gives them broad knowledge, and in-context learning (ICL) lets them use information inside a single chat session. But ICL is forgotten when the session ends. Updating the model with new data can cause catastrophic forgetting, where learning new things wipes out old skills.

Key facts

• The framework aims to let models keep long-context knowledge up to 10M (10 million) tokens.
• The sleep phase uses pre-allocated Mixture-of-Experts (MoE) memory blocks.
• The model updates its internal weights using practice examples it generates itself.
• The core code and model design stay unchanged.

How the sleep process works

The method is inspired by how the human brain stabilizes memory during rest, especially during non-REM slow-wave sleep. The model adds new capacity during sleep by unlocking small expert modules. It then moves fragile short-term memories into these stable, long-term parts. This is done with knowledge distillation, where a student model learns from a teacher, combined with reinforcement learning (RL) style imitation.

Why it matters

If this works, AI systems could keep learning after they ship, instead of staying stuck at a fixed knowledge cutoff date. They could add new facts, remember more, and reason better, without losing what they already know. This could make AI assistants more useful over long periods of time.