AI Insight
This study tested whether learned gate weights in neural posterior estimation can reliably indicate which features of EEG data are important for inferring brain model parameters. When comparing gate-based importance measures against a rigorous ablation benchmark (retraining the model after removing each feature), the researchers found they were negatively correlated, with the most important feature according to ablation being among the most down-weighted by the gates. This contradicts the assumption that learned gates in simulation-based inference networks reveal true feature importance.
Why it matters
The findings suggest that researchers using simulation-based inference methods to fit mechanistic models should not rely on learned gate weights for scientific interpretation without additional validation. This has practical implications for computational neuroscience and other fields where understanding which data features drive model predictions is critical for scientific insight and trust in machine learning pipelines.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Simulation-based inference (SBI) provides a powerful way to fit mechanistic models to data, but offers little insight into which information in the data shapes the inferred parameter values. It could therefore benefit from robust and inexpensive interpretability methods. Learned weighting mechanisms such as feature gates provide a potential route to low-cost interpretability, although their interpretation assumes that predictive training reveals which inputs drive performance. We examined this assumption in a computational neuroscience setting, where neural posterior estimation (NPE) is used to infer four physiological parameters of a dynamical model of interacting neuronal populations (Jansen-Rit). We tested whether the learned gate weights captured feature importance by comparing them with a retrain-based ablation benchmark, in which each feature was removed in turn and the NPE retrained. We considered two jointly trained gates: independent sigmoid weights and competing softmax weights. If these reliably reflect feature importance, they should align with the ablation-based estimates. Across nine independent training seeds, however, gate-based and ablation-based importance were negatively correlated (Spearman’s rho approximately -0.61 for sigmoid gating and -0.56 for softmax gating). The feature that ablation identified as most important, total signal power, was among the features the gate down-weighted most, despite gating costing negligible predictive accuracy. These results extend a pattern previously documented for attention mechanisms in natural language processing and for explainability methods in electroencephalography (EEG) deep learning to the simulation-based inference setting, and suggest caution when interpreting learned gate weights as measures of causal feature importance in scientific machine-learning pipelines.
Source: Gate-Based Feature Importance in Simulation-Based Inference for EEG: A Cautionary Tale