AI Insight
Researchers investigated how different levels of model compression (quantization) affect the ability of large language models to learn new tasks without forgetting previous ones. Contrary to expectations, they found that 8-bit quantized models outperformed higher-precision 16-bit models on sequential tasks by 8-15%, with 4-bit models showing even more dramatic improvements on code generation tasks. The study suggests that quantization introduces beneficial noise that prevents overfitting to new tasks, and that even small replay buffers containing just 0.1-1% of previous training data can significantly improve knowledge retention across all precision levels.
Why it matters
These findings challenge the assumption that higher precision is always better for AI models, showing that compressed models can be both more computationally efficient and better at continual learning. This has practical implications for deploying AI systems that need to learn new tasks over time without expensive retraining, potentially reducing both computational costs and the catastrophic forgetting problem in real-world applications.
Understand the Science
arXiv:2512.18934v2 Announce Type: replace-cross
Abstract: Catastrophic forgetting poses a fundamental challenge in continual learning, particularly when models are quantized for deployment efficiency. We systematically investigate the interplay between quantization precision (FP16, INT8, INT4) and replay buffer strategies in large language models, revealing unexpected dynamics. While FP16 achieves superior initial task performance (74.44% on NLU), we observe a striking inversion on subsequent tasks: quantized models outperform FP16 by 8-15% on final task forward accuracy, with INT4 achieving nearly double FP16’s performance on Code generation (40% vs 20%). Critically, even minimal replay buffers (0.1%) dramatically improve retention – increasing NLU retention after Math training from 45% to 65% across all precision levels – with INT8 consistently achieving the optimal balance between learning plasticity and knowledge retention. We hypothesize that quantization-induced noise acts as implicit regularization, preventing the overfitting to new task gradients that plagues high-precision models. These findings challenge the conventional wisdom that higher precision is always preferable, suggesting instead that INT8 quantization offers both computational efficiency and superior continual learning dynamics. Our results provide practical guidelines for deploying compressed models in continual learning scenarios: small replay buffers (1-2%) suffice for NLU tasks, while Math and Code benefit from moderate buffers (5-10%), with quantized models requiring less replay than FP16 to achieve comparable retention. Code is available at https://github.com/Festyve/LessIsMore.
Source: When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models