AI Insight
This paper identifies that quantizing large language models below 4-bit precision fails because quantization errors accumulate through the residual stream connections between transformer layers, causing the internal representations to drift increasingly from their original distribution. The authors propose RDQ (Residual Distribution Quantization), which compensates for this drift by calibrating each layer's quantization parameters using the actual drifted activations from previously quantized layers rather than the original full-precision activations. Testing on LLaMA-3-8B, Qwen-2.5-7B, and Mistral-7B shows RDQ achieves the lowest perplexity scores for 3-bit and 4-bit quantization compared to existing methods, with improvements up to 46.4% over baseline approaches.
Why it matters
This technique enables more aggressive compression of large language models while maintaining performance, making them feasible to deploy on resource-constrained devices like smartphones and edge computing hardware. The method works with standard quantization formats and adds no computational overhead during inference, making it immediately practical for real-world deployment.
Understand the Science
Abstract: Post-training quantization (PTQ) of large language models degrades sharply below 4-bit precision. We identify the root cause as residual stream distributional drift: quantization noise injected at each transformer layer accumulates in the shared residual representation, causing KL divergence from the FP16 baseline to grow super-linearly with depth (Pearson r=0.999 with log-perplexity, p<0.001, confirmed across all tested methods and bit-widths). We discover that 84% of LLaMA-3-8B layers exhibit non-Gaussian residual distributions (KS test, p<=0.05), and that per-layer residual stream variance grows 6,548x across depth. We propose RDQ (Residual Distribution Quantization), a PTQ framework whose central contribution is Cascaded Error Compensation (CEC): a sequential calibration procedure that captures the actual drifted activations each layer receives (computed by running calibration data through already-quantized upstream layers) and fits per-channel AWQ-style scales against those drifted inputs, with scales folded into preceding RMSNorm weights for exact mathematical equivalence at zero inference overhead. RDQ achieves state-of-the-art results on all three tested architectures: LLaMA-3-8B: 7.55 / 5.62 PPL (W3/W4); Qwen-2.5-7B: 7.46 / 6.38 PPL; Mistral-7B: 6.88 / 5.73 PPL. RDQ beats the best published baseline (LeanQuant/SpinQuant) at every model and bit-width combination, with gains up to -46.4% vs. RTN at W3A16 on LLaMA-3-8B. All output is standard group-128 asymmetric quantization, deployable on Qualcomm AIMET, GGUF, and any standard inference stack at zero runtime overhead.
Source: RDQ: Residual Distribution Quantization for Large Language Models