AI Insight
This paper introduces a new method for interpreting neural networks by inverting internal features back to the input space while preserving information about the original input that generated those features. Unlike traditional feature inversion methods that search for any input matching a target feature representation, this approach grounds the inversion in the local network geometry at the source input using backpropagation signals repaired through Wiener filtering. The method works across different neural network architectures (CNNs and Transformers) without requiring optimization for each query, enabling systematic visualization of what individual layers and channels extract from specific inputs.
Why it matters
This technique provides a more reliable way to understand what neural networks learn and how they make decisions by linking internal representations directly to the inputs that created them. It could improve model debugging, safety verification, and trust in AI systems by making the feature hierarchy of deep networks inspectable at the level of individual components.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Interpreting a neural network requires understanding what its internal features extract from a particular input. Feature inversion seeks to express a selected feature in the input domain, but canonical iterative methods search for an input whose re-encoded representation matches the target. Because many inputs can satisfy this constraint, target matching alone does not specify the inverse associated with the sample that generated the feature. We formulate source-grounded feature inversion by conditioning the inverse on the source-local network geometry at the target-generating input. At each boundary of the computational DAG, backpropagation provides the correct reverse dependencies but transports an adjoint signal rather than an upstream-state estimate. We locally repair this signal with a closed-form matrix Wiener map from a mean-seed VJP to the upstream state, followed by a second Wiener map for the JVP forward-consistency residual, and compose the repaired states through the same DAG in one finite reverse pass. One calibrated zero-intercept map family supports new inputs, depths, channels, and channel groups across diverse CNN and Transformer architectures, tensor components, and visual distributions without query-specific optimisation. Matched target and source controls verify that each inverse depends on the selected feature and the local operators of the sample being explained, rather than a target-independent image template. Prediction-conditioned feature atlases align these visualisations with independent interventions on the corresponding internal features. Together, source-grounded feature inversion opens the model’s hidden feature hierarchy to inspection at the level of individual layers and channels, linking what the network extracts from an input to the internal evidence that shapes its decision.
Source: From Preimage Search To Source-Grounded Feature Inversion