AI Insight
Researchers developed Daedalus-150M, a 150-million parameter language model specifically designed for efficient CPU inference by combining convolution and attention mechanisms. The hybrid architecture uses full attention in only 6 of 18 blocks while the remaining 12 use short convolutions with fixed memory, resulting in 1.76-2.08x faster decoding at longer contexts compared to traditional all-attention models. Trained on 59.9 billion tokens, it outperforms models like GPT-2 124M and Pythia-160M that were trained on 3-6 times more data, achieving a score of 47.31 on a five-task benchmark.
Why it matters
This architecture-first approach demonstrates that language models can be purpose-built for resource-constrained environments rather than simply compressed after training. The model's efficiency gains increase with context length, making it particularly suitable for CPU-based deployment in applications where GPUs are unavailable or cost-prohibitive.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Small language models are usually built like large ones and then squeezed onto a CPU afterwards. We did the opposite: we fixed the target first, one user, one token at a time, 4-bit weights, ordinary CPU, and chose the architecture to suit it. The result keeps full attention in only 6 of its 18 blocks. The other 12 use short convolutions whose memory is two timesteps wide no matter how long the conversation gets, so two thirds of the network never re-reads a growing cache.
Trained from scratch on 59.9B tokens, the model scores 47.31 on a five-task benchmark against a bar of 42.20 that was fixed before training began. It beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-neo-125M, all trained on three to six times more data, and exceeds MobileLLM-125M’s published score despite that model seeing a trillion tokens. Validation bits-per-byte is 0.8685.
To check the architecture rather than the training recipe, we trained a conventional all-attention model of the same size on the same data, and wrote down the winning condition before scoring either. The hybrid won the chosen quality metric by 0.81%, matched it on downstream tasks, produced a 6.3% smaller 4-bit file, and decoded 1.76x faster at 2048 tokens of context, 2.08x against an external model of similar size. In every measurement the speed advantage is near zero at an empty context and grows with length, which is what the mechanism predicts and what a merely leaner model would not show. A simple bandwidth calculation predicts only 1.17x, so memory volume alone does not explain the gap.
We also report what did not work: an unmitigated 4-bit quality cost, roughly half the convolution channels ending up inert and impossible to remove, and a vocabulary larger than this model size warrants.
Source: Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference