AI Insight
This study introduces a new evaluation framework for quantized Large Language Models that measures information loss by comparing the full probability distributions of predictions, rather than relying solely on accuracy metrics. Through 120 experiments across five model architectures and four reasoning benchmarks, the researchers demonstrate that traditional accuracy measures can miss substantial degradation in model behavior during compression, while divergence-based metrics like Jensen-Shannon Divergence reveal consistent patterns of fidelity loss. The findings show that mixed-precision quantization schemes generally preserve distributional fidelity better than uniform quantization at comparable memory savings.
Why it matters
As AI models are increasingly deployed on resource-constrained devices, this work provides practitioners with better diagnostic tools to evaluate model compression quality beyond simple accuracy scores. The framework could help prevent deployment of seemingly accurate but distributionally unstable quantized models, improving reliability of edge-deployed language models.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Deployment of Large Language Models (LLMs) on memory-constrained edge devices relies heavily on aggressive post-training quantization. However, evaluating these models is largely based on zero-shot task accuracy, which depends solely on argmax predictions and is insensitive to changes in the underlying predictive distribution. Consequently, accuracy can exhibit unstable, non-monotonic behavior under progressive quantization, masking substantial fidelity loss relative to the BFloat16 (BF16) uncompressed base model and providing misleading deployment signals. We introduce a distribution-sensitive evaluation framework quantifying information loss in quantized LLMs as the divergence between full-vocabulary predictive distributions at the token decision boundary. We compute statistical distances, including Jensen-Shannon Divergence and Total Variation Distance, between outputs of full-precision and quantized models, enabling a fine-grained analysis of distributional shift. Using this framework, we quantify probability mass displacement and distributional drift relative to the BF16 reference, capturing predictive distribution changes not reflected in top-1 accuracy. We conduct a 120-run experimental matrix across five foundation architectures and four reasoning benchmarks under progressive quantization regimes, from uncompressed BF16 to Q2_K, providing a systematic fidelity analysis. Our results show divergence metrics generally increase under stronger quantization, complementing task accuracy with a fidelity signal. Across tested llama-cpp schemes, mixed-precision Q4_K generally yields lower divergence than uniform Q4_0 at similar memory footprints. These findings motivate distribution-aware evaluation as a practical diagnostic complement to task accuracy; they do not directly establish correctness, calibration, safety, or user-perceived quality.
Source: Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs