AI Insight
This study identifies a position bias problem in On-Policy Distillation (OPD), a reinforcement learning technique where a student model learns from a teacher model through token-level supervision. The researchers found that tokens generated later in a sequence receive degraded supervision because the student's outputs increasingly deviate from the teacher's distribution over time, with the first 30% of tokens providing comparable learning to using all tokens while the last 30% contribute minimally. To address this, they developed Importance-Weighted On-Policy Distillation (IW-OPD), which automatically assigns higher weights to earlier tokens and lower weights to later ones based on accumulated distribution discrepancy, resulting in faster convergence and up to 6.9 point improvement on AIME-2025 benchmarks.
Why it matters
This work could significantly improve the efficiency of training large language models and AI systems through distillation, reducing computational costs and training time while achieving better performance. The findings have practical applications for developing more capable AI systems in mathematical reasoning and other domains requiring sequential decision-making.
Understand the Science
arXiv:2606.22600v3 Announce Type: replace-cross
Abstract: On-Policy Distillation (OPD) improves the learning efficiency of standard reinforcement learning through dense, token-level supervision from teachers. In the standard KL objective of OPD, token-level losses are uniformly averaged, implying equal weights for all tokens. However, we discover that not all tokens are created equal: as student rollouts grow longer, they deviate further from the teacher’s distribution, leading to degraded supervision quality at later positions. As a result, OPD using only the first 30% of tokens can perform comparably to using all tokens, whereas OPD using only the last 30% of tokens barely learns anything. In this work, we provide a principled understanding of this issue through the lens of constrained optimization. Based on these insights, we derive Importance-Weighted On-Policy Distillation (IW-OPD), in which the weight assigned to each token depends on the accumulated discrepancy between the student’s and teacher’s distributions, naturally upweighting earlier tokens and downweighting later ones with larger deviations. We show that IW-OPD converges significantly faster than OPD, with better learning efficiency, and achieves better final performance than standard OPD in both same-size and cross-scale settings, improving performance up to 6.9 points on AIME-2025.