AI Insight
This paper introduces HyenaND, a computational operator that processes multi-dimensional data (images, volumes, medical scans) more efficiently than traditional attention mechanisms while maintaining global context awareness. The method achieves subquadratic computational complexity (O(L log L)) through input-dependent convolutions that respect the native spatial structure of data, avoiding the need to flatten multi-dimensional inputs into arbitrary one-dimensional sequences. Testing across genomics, computer vision, medical imaging, and physics simulations shows that HyenaND matches attention-based models in accuracy while offering computational speedups, with hybrid approaches combining both methods performing best.
Why it matters
This work addresses a key bottleneck in processing large-scale multi-dimensional data like medical images and scientific simulations, where standard attention mechanisms become prohibitively expensive. The computational efficiency gains could enable real-time analysis of high-resolution medical scans and more complex scientific modeling without sacrificing accuracy.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Subquadratic alternatives to attention require compromises when applied to multi-dimensional data: standard convolutions lack global receptive fields and input dependency, while recurrent models require rasterizing data such as images, volumes, and partial differential equation (PDE) into an ad-hoc $1rm D$ scan order that violates their spatial structure. We introduce textit{HyenaND}, a subquadratic, global, input-dependent operator that acts directly on the native geometry of multidimensional data through convolutions with implicitly parametrized global, input-dependent multi-dimensional convolutional kernels. Our CUDA implementation, texttt{nSubQ}, fuses the FFT-convolution path to turn HyenaND’s $mathcal{O}(L log L)$ scaling into wall-clock speedups. Across long-context genomics, computer vision, medical imaging, and PDE modeling, pure HyenaND stacks match the accuracy of strong attention baselines, while hybrid configurations that interleave HyenaND and attention layers outperform both pure attention and strong recurrence-based hybrids.
Source: Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions