AI Insight
This paper introduces Quantized approximately Orthogonal Recurrent Neural Networks (QORNNs), which compress Orthogonal RNNs by reducing the precision of weights and activations while maintaining their ability to handle long-term dependencies. The researchers developed two training strategies combining quantization-aware training with orthogonal projections to create stable quantized models. The resulting fully-quantized networks are the only current models capable of solving the Copy-task with 1000 timesteps and outperform existing quantized models on tasks requiring long-term memory, though they remain less effective than LSTMs on certain tasks.
Why it matters
This work enables deployment of powerful recurrent neural networks on resource-constrained devices like IoT sensors and mobile phones by significantly reducing memory and computational requirements. The ability to perform pure integer computation while maintaining performance on long-sequence tasks opens possibilities for real-time processing in edge computing applications where full-precision models are impractical.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
-cross
Abstract: In recent years, Orthogonal Recurrent Neural Networks (ORNNs) have gained popularity due to their ability to manage tasks involving long-term dependencies, such as the Copy-task, and their linear complexity. However, existing ORNNs utilize full precision weights and activations, which prevents their deployment on compact devices. In this paper, we explore the quantization of the weight matrices in ORNNs, leading to Quantized approximately Orthogonal RNNs (QORNNs). The construction of such networks remained an open problem, acknowledged for its inherent instability. We propose and investigate two strategies to learn QORNN by combining quantization-aware training (QAT) and orthogonal projections. We also study post-training quantization of the activations for pure integer computation of the recurrent loop. The most efficient fully-quantized models are currently the only alternatives for solving the Copy-task with 1000 timesteps. They also surpass existing quantized models on tasks requiring long-term dependencies, such as permuted MNIST as well as some Internet of Things (IoT) tasks. However, they are less effective on tasks where long short-term memory (LSTMs) and other gated RNNs outperform ORNNs.
Source: On Orthogonal Recurrent Neural Networks Quantization