AI Insight
This paper introduces FAIR-Calib, a calibration framework for quantizing Diffusion Large Language Models (dLLMs) to reduce computational costs while maintaining performance. The researchers identified that standard post-training quantization causes errors at decision boundaries ("frontiers") where token commitments are made, and these errors propagate irreversibly through the generation process. Their two-stage method uses a position-aware weighting scheme to protect vulnerable frontier states during layer-wise calibration, achieving superior performance on 4-bit weight and activation quantization compared to existing approaches on LLaDA and Dream models.
Why it matters
Efficient quantization of diffusion language models could enable deployment of these advanced models on resource-constrained devices while reducing energy consumption and inference costs. This technique specifically addresses a fundamental challenge in quantizing models with iterative refinement mechanisms, potentially accelerating the practical adoption of diffusion-based language generation systems.
Understand the Science
arXiv:2606.06547v2 Announce Type: replace
Abstract: Diffusion Large Language Models (dLLMs) refine tokens iteratively but commit them irreversibly, leading to a “stability lag” where early decisions remain fragile even after being written. We reveal that Post-Training Quantization (PTQ) error easily flips these borderline decisions at the write frontier, which are then permanently locked in and amplified. To address this, we propose Frontier-Aware Instability-Reweighted Calibration (FAIR-Calib), a two-stage PTQ framework for dLLMs. Stage I probes a full-precision teacher to estimate a position prior that combines frontier hits and masked-stage reliability. Stage II performs off-policy, layer-wise calibration by minimizing a reweighted hidden-state MSE, effectively prioritizing the protection of fragile frontier states without requiring expensive end-to-end diffusion rollouts. We further theoretically justify our weighted objective as a surrogate for output KL divergence. Empirically, FAIR-Calib consistently outperforms state-of-the-art baselines on LLaDA and Dream (W4A4), significantly reducing frontier decision flips and suppressing post-commit mismatches across diverse benchmarks.