AI Insight
This study introduces a framework called EARL (Emergent Adaptive Representation Learning) that identifies two distinct modes of compression in neural network learning. The researchers found that parameter-space compression, where networks with different initial conditions converge to similar representations, occurs early in training and independently of the specific task. In contrast, state-space compression, where representations gain predictive power about their own future states, emerges later (around 30 epochs) and only when the network learns task-specific temporal structure. These findings were demonstrated using recurrent reservoir networks with trained bottlenecks.
Why it matters
This research helps explain why neural networks, both biological and artificial, develop similar internal representations despite differences in their underlying architecture and initialization. Understanding these two phases of learning could inform better neural network design and training strategies, and provide insights into how biological brains might develop invariant representations.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Learned representations show striking invariance across biological and artificial neural networks, with different substrates and variations driven by drift, heterogeneity and initialisation. To understand this invariance, we propose Emergent Adaptive Representation Learning (EARL), a framework that distinguishes two modes of emergence in learned representations: state-space compression, where a learned representation gains predictive power over its own future beyond what the network’s units can predict individually, and parameter-space compression, where networks with different micro-level parameters converge to the same representation. We formalise both as Fisher sensitivities and introduce ensemble-level information-theoretic measures for each. In a recurrent reservoir with a trained bottleneck, parameter-space compression rises within the first few epochs of training and partly independently of the task, whereas state-space compression rises about 30 epochs later and only when the network begins to learn the temporal structure of the task. These results suggest that invariance to the substrate and task-specific emergent structure are acquired in distinct phases of learning.
Source: Invariance before structure: two modes of emergent compression in representation learning