AI Insight
This paper introduces HeRo-Q, a novel post-training quantization method that addresses the "low error, high loss" problem in large language model compression by conditioning the Hessian matrix of the loss landscape. The algorithm applies a learnable rotation-compression transformation to weights before quantization, which reduces sensitivity in high-curvature directions and improves robustness to quantization noise. Experimental results on Llama and Qwen models demonstrate superior performance compared to existing methods like GPTQ and AWQ, particularly in ultra-low bit settings such as 3-bit weights, where it achieves 70.15% accuracy on GSM8K with Llama3-8B while avoiding model degradation.
Why it matters
This work enables more aggressive compression of large language models without catastrophic performance loss, making powerful AI models more accessible for deployment on resource-constrained devices. The method's compatibility with existing quantization pipelines and negligible computational overhead suggests practical applicability for reducing the storage and inference costs of deploying large language models.
arXiv:2601.21626v2 Announce Type: replace
Abstract: Post Training Quantization (PTQ), a mainstream model compression technique, often leads to the paradoxical ‘low error, high loss’ phenomenon because it focuses solely on minimizing quantization error. The root cause lies in the Hessian matrix of the LLM loss landscape: a few high curvature directions are extremely sensitive to perturbations. To address this, we propose the Hessian Robust Quantization (HeRo Q) algorithm, which applies a lightweight, learnable rotation-compression matrix to the weight space prior to quantization. This joint framework reshapes the loss landscape by reducing the largest Hessian eigenvalue and reducing its max eigenvalue, thereby significantly enhancing robustness to quantization noise. HeRo-Q requires no architectural modifications, incurs negligible computational overhead, and integrates seamlessly into existing PTQ pipelines. Experiments on Llama and Qwen models show that HeRo Q consistently outperforms state of the art methods including GPTQ, AWQ, and SpinQuant not only achieving superior performance under standard W4A8 settings, but also excelling in the highly challenging W3A16 ultra low bit regime, where it boosts GSM8K accuracy on Llama3 8B to 70.15% and effectively avoids the logical collapse commonly seen in aggressive quantization.
Source: HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioning