AI Insight
Researchers developed Ventaglio, a hardware extension for vector processors that accelerates sparse tensor operations in AI Transformer models by exploiting both weight and activation sparsity. The system achieves 6.9-7.4x speedup on sparse tensor operations with only 3.1% additional hardware overhead, and demonstrates 2.4-5.25x faster inference on a pruned LLaMA-3-8B model with 40-60% sparsity compared to dense computation baselines.
Why it matters
This work addresses a critical bottleneck in running large language models efficiently by providing specialized hardware support for sparse computations. The minimal area overhead makes it practical for integration into existing vector processors, potentially enabling faster and more energy-efficient deployment of AI models in resource-constrained environments.
Understand the Science
Abstract: Fine-grained weight pruning and activation sparsification have emerged as effective approaches for reducing the compute and memory cost of inference for Transformer models. In the moderate-sparsity regime, Gustavson’s dataflow provides a natural execution model for exploiting both activation and weight sparsity on vector processors through metadata-driven indexed accumulation. However, existing RVV architectures lack native support for this pattern, forcing kernels to rely on software index decoding and L1-backed indexed memory operations that keep sparse tensor contractions far below their roofline performance bound. We present Ventaglio, a runtime-configurable sparse execution unit coupled with RVV ISA extensions that drives sparse tensor contractions toward their roofline through indexed gather-accumulate-scatter support. Integrated into an open-source vector processing cluster and implemented in 12nm FinFET, Ventaglio accelerates sparse tensor contraction kernels by $6.9text{–}7.4times$ over optimized RVV baselines, with only $3.1%$ area overhead for a cluster of tightly-L1 coupled vector processing elements. We build a performance-accurate instruction-level model of the Ventaglio extension, calibrate it against RTL implementation, and leverage it for scale-out performance analysis on a large $4times4$ multi-cluster system. Using a DuoGPT-pruned LLaMA-3-8B model with practical $40text{–}60%$ dual sparsity, Ventaglio achieves $2.40text{–}5.25times$ and $2.06text{–}3.16times$ speedup over dense baselines during prefill and autoregressive decoding, respectively.
Source: At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference