AI Insight
This study investigates how reinforcement learning (RL) enables large language models to compositionally generalize by combining known skills in novel ways, using a mathematical "Countdown" task as a test case. Researchers analyzed solution patterns as expression trees and found that models successfully generalize to out-of-distribution problems, but skill acquisition follows a predictable order—mastering balanced tree structures before unbalanced ones, with persistent difficulty on right-heavy structures. The findings reveal that RL post-training induces genuine compositional skill reuse beyond simple length generalization, though certain compositional structures remain challenging.
Why it matters
Understanding how RL teaches models to compose skills has direct implications for developing more capable AI reasoning systems, particularly for complex tasks like mathematical problem-solving and formal theorem proving. The identification of structural patterns that facilitate or hinder skill transfer could inform more efficient training strategies for AI systems across domains requiring compositional reasoning.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: While reinforcement learning (RL) successfully enhances reasoning in large language models, its role in fostering compositional generalization (the ability to synthesize novel skills from known components) is often conflated with mere length generalization. To this end, we study what RL post-training teaches about skill composition and how the composition structure affects the skill transfer. We focus on the Countdown task (given n numbers and a target, form an expression that evaluates to the target) and analyze model solutions as expression trees, where each subtree corresponds to a reusable subtask and thus can be viewed as a “skill.” Tracking tree shapes and their success rates over training, we find: (i) out-of-distribution (OOD) generalization to larger n and to unseen tree shapes, indicating compositional reuse of subtasks; (ii) the order of skill acquisition depends upon the composition structure—models master shallow balanced trees (workload is balanced between subtasks) before deep unbalanced ones, with persistent fragility on right-heavy structures for a fixed composition depth. Our diagnostic reveals what is learned, in what order, and where generalization fails, clarifying how RL-only post-training induces OOD generalization beyond what standard metrics such as pass@k reveal. To study the implications of our findings for real-world tasks, we present a preliminary study on formal theorem proving.
Source: How Does RL Post-training Induce Skill Composition? A Case Study on Countdown