AI Insight
This study evaluates a lattice-based pruning method for recurrent neural networks that attempts to preserve information flow between neurons by modeling dependencies as mathematical structures. Testing on LSTM networks with MNIST data revealed that the theoretically-motivated approach performed no better than simpler methods like random pruning or magnitude-based selection, and imposing ordered structure from initialization significantly reduced accuracy at high sparsity levels. The findings suggest that meaningful hierarchical order in recurrent networks must be artificially imposed rather than discovered through dependency analysis.
Why it matters
The research challenges assumptions about how to efficiently compress brain-inspired neural networks, indicating that computationally expensive graph-theoretic approaches offer no advantage over simpler pruning methods. This could save researchers time by steering them toward more practical compression techniques for deploying recurrent networks on resource-constrained devices.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Magnitude pruning of recurrent neural networks (RNNs) ignores how information flows between neurons. An earlier version proposed to model RNN dependencies as a partially ordered set (poset) and to retain the meet-irreducible elements of its dependency lattice. By Birkhoff’s representation theorem, the meet-irreducible ideals of a finite poset are in bijection with its elements, so retaining them retains everything; the critical-set criterion used earlier selects precisely the elements comparable to all others, of which a dense recurrent layer has none. Condensing a thresholded dependency graph into strongly connected components yields a well-defined dependency order, a graded order score and an acyclic pruning rule; for Cohen-Grossberg networks and LSTMs, acyclic pruning makes local stability decompose over modules. For a 128-unit LSTM pruned at matched sparsity on row-wise and chunked sequential MNIST (three seeds, equal training budgets), the original rules remove every recurrent weight or equal random column pruning; the order score is no better than degree, PageRank or norm; trained weights show little directional bias; and imposing acyclicity from initialisation costs 2.1 and 10.5 percentage points at 50% and 90% sparsity. In the networks studied, order structure must be imposed rather than discovered.