AI Insight
Maglev is a new recurrent Transformer architecture that maintains fixed-size memory while enabling parallel training through a dual-model approach. The system uses a "prefiller" model with full attention to generate memory targets and a "decoder" model with sliding-window attention that learns to match these targets through a memory consistency loss, allowing efficient inference using only the decoder. Experiments show Maglev outperforms existing sliding-window and latent recurrent transformer baselines on validation loss and pretraining benchmarks.
Why it matters
This architecture addresses a key limitation in processing long sequences with Transformers by combining the computational efficiency of sliding-window attention with the benefits of recurrent memory, potentially enabling more efficient language models that can handle longer contexts without proportionally increasing computational costs during inference.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: We introduce ours{}, a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. ours{} consists of two coupled models: a prefiller $Q$, which leverages full attentionfootnote{In practice, we use interleaved full and sliding-window attention for $Q$, as this yields stronger performance. The essential requirement is that $Q$ be more expressive than $P$, with access to the full history.} to produce memory targets $m’_t$, and a decoder $P$, which uses only sliding-window attention and recurrent K/V injection to produce decoder memories $m_t$ for next-token prediction. We train ours{} with a memory consistency loss that aligns $m_t$ with $m’_t$, allowing inference to use $P$ alone. Empirically, ours{} improves validation loss and downstream pretraining benchmarks over sliding-window and latent recurrent transformer baselines. Moreover, sharing parameters between $P$ and $Q$ reduces parameter memory while preserving most of the gains.
Source: Maglev: Sliding Recurrent Memory