AI Insight
This paper proposes a theoretical connection between randomness and compression in neural networks by measuring Gibbs entropy over compressed model weights. The authors test this proposition through iterative compression-training cycles on a computer vision task, using three compression methods: random pruning, magnitude pruning, and a novel dual tomographic compression approach. They demonstrate a comonotonic relationship between compression ratio and task performance, suggesting that entropy measurements on compressed models can quantify the trade-off between model size and capability.
Why it matters
Understanding the fundamental relationship between model compression and performance could lead to more principled approaches for creating efficient neural networks. This has practical implications for deploying machine learning models on resource-constrained devices and could help optimize the balance between model size and accuracy in real-world applications.
Understand the Science
arXiv:2505.23869v5 Announce Type: replace-cross
Abstract: A proposition that connects randomness and compression is put forward via Gibbs entropy over set of measurement vectors associated with a lossy compression process. In building this connection, we use a performance of a learning task as a probe of compression in iterative compress-train cycles. This can be thought as iterative coarse-graining from statistical mechanics perspective using thermodynamic efficiency as a probe. We formulate this connection via comonotonic relationship within a very small decrease in compression ratio and the performance. We have showcase the validity of this proposition with a canonical vision task in deep learning with three different model compression processes as {it a baseline model}. We use the following, simpler to more complex model compression approaches: (1) random pruning,(2) magnitude pruning, and (3) a more complex compression by using dual tomographic compression, which utilizes compressed sensing in dual fashion which is introduced as a new method. We use remaining weights of deep learning network as a measurement vector where we measure the Gibbs entropy. We show case the idea that there is an inherent computable connection between compression probed by performance and randomness from an entropy measure on the learned model.