AI Insight
Researchers developed CHIVE (Counterfactual Hypothesis Investigation Via Edits), an automated system that tests AI explanations by evaluating whether they can predict how language models behave when inputs are modified. The study found that common interpretability techniques did not improve the ability to predict model behaviors on counterfactual inputs, but training models on CHIVE-generated data improved their ability to predict outcomes in new situations. CHIVE automatically discovers and tests explanations for naturally-occurring behaviors in large language models by generating thousands of counterfactual experiments.
Why it matters
This work provides a rigorous framework for evaluating whether AI explanations are actually useful rather than merely plausible-sounding. The findings suggest current interpretability methods may be less reliable than assumed, which has important implications for AI safety, transparency, and our ability to understand and predict AI system behaviors.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Many areas of AI research, such as language model interpretability and chain of thought faithfulness, seek to explain model behaviors. But what constitutes a “good” explanation? In this work, we evaluate explanations through the lens of counterfactual simulatability-whether the explanation is useful for predicting model behaviors on related counterfactual inputs. To this end, we introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), a novel agentic pipeline that identifies unexpected model behaviors in the wild and investigates them with counterfactual prompt edits. This yields thousands of high-quality explanations for naturally-occurring model behaviors along with supporting counterfactual evidence. We apply CHIVE in two ways. First, we evaluate whether common LLM interpretability techniques improve an agent’s ability to predict counterfactual model behaviors. Surprisingly, we find no uplift from any of the interpretability techniques studied. Second, we use CHIVE to generate training data. We find that training models to predict outcomes of CHIVE-generated counterfactual experiments generalizes to various out-of-distribution settings. Overall, CHIVE automatically discovers explanations of naturally-occurring LLM behaviors, enabling us to evaluate and improve methods for explaining LLM behaviors.