AI Insight
Researchers investigated whether clustering attention heads in neural networks based on co-activation patterns actually identifies functional circuits. Testing across multiple billion-parameter language models, they found that while clustering identifies statistically meaningful groups in standard dense models, these groups must be validated through causal ablation tests to confirm they operate as genuine circuits. In one mixture-of-experts model, statistically significant clusters failed this validation test, demonstrating that co-activation patterns alone are insufficient to confirm functional circuits.
Why it matters
This work establishes critical methodology for interpretability research in AI systems, showing that statistical patterns in neural networks require causal validation before being accepted as functional explanations. The findings will help researchers avoid false conclusions when trying to understand how large language models actually process information internally.
Understand the Science
Abstract: Interpretability increasingly treats groups of components, not individual units, as the basic object, and proposes to find them by clustering co-activation statistics. We ask whether such a cheap signal actually identifies an attention-head circuit. Adapting a sparse-autoencoder clustering recipe to attention heads — but validating by causal ablation rather than reconstruction — we cluster heads and then run a closure test: ablate the discovered community and compare per-example damage to matched-random controls. Across two dense 1B-scale models (Pythia 1B, OLMo 1B) and two input distributions, the communities pass closure. In a Mixture-of-Experts model (OLMoE-1B-7B), route-conditional clustering recovers a statistically real signal that nonetheless does not survive closure — ablation improves loss, the wrong direction. Extending closure across training, attention-target selectivity and participation ratio decouple from function in both directions. We conclude that a cheap signal is a circuit proposal, not a confirmed circuit; closure is what separates them.
Source: Closure-Validated Circuit Discovery in Attention Heads: Co-activation Proposes, Ablation Disposes