AI Insight
SORT (Selective Off-policy Reference Tuning) is a new reinforcement learning method that addresses a critical limitation in GRPO-style approaches where training stalls when all generated responses to difficult prompts fail. The technique extracts a plan from reference solutions and selectively upweights tokens that become more predictable when conditioned on this plan structure, converting complete failures into useful learning signals without modifying the rollout generation process. Testing across three model architectures and eight reasoning benchmarks demonstrates consistent improvements over GRPO and guidance baselines, with the most substantial gains observed in weaker models.
Why it matters
This approach could significantly improve the training efficiency of AI reasoning systems by enabling learning from difficult examples that would otherwise be discarded, potentially reducing the computational resources and data requirements needed to develop capable reasoning models.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Reinforcement learning with verifiable rewards helps reasoning, but GRPO-style methods stall on hard prompts where all sampled rollouts fail. SORT adds a repair update for those failures without changing rollout generation: it derives a plan from the reference solution, compares token probabilities with and without that plan, and gives higher weight to tokens that become more predictable under plan conditioning. This turns all-wrong prompts into selective, structure-aware learning signals instead of uniform imitation. Across three backbones and eight reasoning benchmarks, SORT improves over GRPO and guidance baselines, with largest gains on weaker models.
Source: Selective Off-Policy Reference Tuning with Plan Guidance