AI Insight
This study examines four consistency training methods designed to prevent language models from changing answers based on unreliable cues. Testing on Gemma-2-2B-it and Llama-3.1-8B-Instruct models revealed that all four methods reduce cue influence but do so indiscriminately, weakening both correct and incorrect cues equally. The fixes are position-dependent, working primarily when cues appear before answer options as in training, and the models fail to learn when cues actually provide valid information.
Why it matters
The findings reveal a significant limitation in current consistency training approaches for AI safety. These methods suppress cue-following behavior without teaching models to discriminate between helpful and misleading information, which has important implications for developing language models that can appropriately weigh different types of evidence in their reasoning.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Language models often change answers because of a cue, a hint such as “I believe the answer is B” that gives no valid evidence. Consistency training (CT) trains a model to answer the same with and without the cue. Four CT methods apply this idea to output tokens (BCT), residual activations (ACT), attention patterns (AttCT) and sampled-answer rates (RMCT). Prior work suggests some learn different solutions. We train all four on Gemma-2-2B-it and Llama-3.1-8B-Instruct with the same data. We test predictions written in advance and inspect the models with a known circuit that carries cues to the answer. All four make the trained cue less effective, and their fixes share three properties. First, correct cues lose about as much influence as wrong ones. Second, the fix is usually large for cues before the answer options, as in training, and much smaller after them. Llama RMCT depends least on position. Third, the circuit heads still carry most remaining cue-induced errors. Making BCT targets depend on the cue’s correctness mostly weakens the fix. CT thus weakens the trained cue without teaching models when a cue is right. Evaluations with only wrong cues at the trained position miss the first two properties.
Source: Consistency Training Suppresses Cue Following Without Teaching Cue Selectivity