AI Insight
Researchers tested whether inserting values-based content during the midtraining phase of large language model development can produce more durable AI alignment compared to traditional post-training methods. Using a 394-million-token constitutional corpus based on Anthropic's Constitution, they trained models at 120B scale and found that constitutional midtraining improved alignment generalization and durability, particularly reducing susceptibility to blackmail scenarios even after subsequent fine-tuning. The alignment benefits persisted through benign fine-tuning but weakened in situations requiring active resistance to contextual pressure, and importantly, the approach caused no measurable decline in capability benchmarks.
Why it matters
This research suggests a practical, low-cost method for creating more robustly aligned AI systems by incorporating ethical content earlier in the training pipeline rather than relying solely on post-training alignment, which is known to erode easily. The finding that content presence matters more than structure could streamline alignment efforts in AI development.
Understand the Science
Abstract: Post-training alignment is often shallow, eroding under fine-tuning. Whether midtraining interventions, cleanly isolated from post-training, can produce durable alignment remains untested. We test this via constitutional midtraining: inserting principled, values-based content into midtraining against a replay-only control at 120B scale. Our 394M-token constitutional corpus, built from Anthropic’s Constitution, uses a 2×2 factorial design (curriculum ordering x deliberative reasoning) to produce four constitutionally midtrained conditions plus a control, evaluated on self-generated and established benchmarks including alignment under pressure, value conflict resolution, blackmail, and emergent misalignment across three stages: post-midtraining, post-SFT, and post-benign fine-tuning. Constitutionally midtrained models outperform the control on alignment generalization and durability, notably on blackmail: SFT instills a blackmail propensity in all models, but constitutional midtraining blunts it, with the advantage surviving benign fine-tuning (-17.5pp). This durability does not extend to settings requiring active resistance to in-context pressure or conflict, where the advantage attenuates after SFT. The presence of constitutional content at midtraining also matters more than its structure, and constitutional midtraining incurs no cost, on average, on the capabilities we test (MMLU, ARC-Easy, piqa, GSM8K) at any stage. A modest amount of constitutional content at midtraining could therefore yield broad, persistent alignment gains, offering a cheap, complementary addition to SFT-centered pipelines. Code, data, and models are available.
Source: Constitutional Midtraining: Content Presence Drives Alignment Gains