AI Insight
This paper introduces tPC-RTRL, a novel learning algorithm that combines Temporal Predictive Coding with Real-Time Recurrent Learning to enable neural networks to learn long-range temporal dependencies while maintaining computational properties suitable for neuromorphic hardware. The method achieves performance nearly identical to traditional backpropagation-through-time on language modeling, machine translation, and drone control tasks, while offering advantages in spatial and temporal locality that are valuable for efficient hardware implementation. The authors prove that under certain assumptions, their approach recovers exact gradients of backpropagation-through-time, and demonstrate that the same inference mechanism can be reused at deployment to incorporate real-time observations.
Why it matters
This work addresses a key challenge in deploying recurrent neural networks on edge devices and neuromorphic hardware by providing a biologically plausible and hardware-efficient alternative to standard training methods. The ability to unify learning and real-time state estimation within the same framework has practical applications for robotics and autonomous systems operating with limited computational resources.
Understand the Science
arXiv:2602.18131v2 Announce Type: replace
Abstract: Temporal Predictive Coding provides a layer-local, parallelisable mechanism for learning in recurrent systems, making it an attractive candidate for online local learning on neuromorphic and edge hardware. However, its recurrent parameter update captures only local temporal relationships, neglecting the historic influence of parameters along the latent-state trajectory, and therefore struggles to assign credit over longer temporal horizons. This work combines for the first time Temporal Predictive Coding with Real-Time Recurrent Learning (tPC-RTRL), incorporating an online influence matrix that tracks this historic effect whilst preserving the spatial and temporal locality properties valued by neuromorphic implementations. Under explicit assumptions, we prove that tPC-RTRL recovers the gradients of backpropagation-through-time exactly. Empirically, a near-equivalence holds across several tasks of varying scale and complexity, including byte-level language modelling on WikiText-103 (tPC-RTRL vs. BPTT: 1.865 vs. 1.864 validation BPC), English–French translation on a CCMatrix subset (20.23 vs. 20.29 BLEU), and a realistic nanodrone system-identification benchmark (0.506m vs. 0.505m mean position error). Finally, we show that the iterative inference mechanism used during training can be reused at deployment time to incorporate intermittent state observations, halving final-position error relative to open-loop rollout on the nanodrone task (0.402m vs. 0.805m) and suggesting a path towards unifying learning and filtering within the same computational framework.
Source: Learning Long-Range Dependencies with Temporal Predictive Coding