AI Insight
This paper introduces Nash-MOPD, a new method for training AI models that combines knowledge from multiple specialized teacher models into a single student model. Unlike previous approaches that use fixed ratios or favor domains showing recent improvement, Nash-MOPD uses game theory principles to dynamically allocate training resources, giving more examples to domains that need them most for comparable progress. The method achieves superior performance on mathematics and code tasks, reaching baseline final scores earlier in training and leading in eight of nine reported metrics across two model architectures.
Why it matters
This technique could enable more efficient development of general-purpose AI systems by better balancing trade-offs when learning multiple specialized skills simultaneously. The approach addresses a fundamental challenge in machine learning: how to allocate limited training resources across different tasks without neglecting domains that learn more slowly.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Multi-teacher on-policy distillation (MOPD) aggregates the capabilities of multiple domain-expert teachers into a single student model. However, all domains in MOPD share the student’s finite budget of optimization steps and rollouts. Fixed uses constant domain sampling ratios based on corpus sizes; by contrast, D$^3$-MOPD adjusts the ratios using the teacher-student reverse-KL divergence and its recent decrease. However, favoring large recent decreases can direct more training toward domains that already improve quickly, while domains that improve more slowly may receive too few examples. To address inflexible allocation and the risk of giving insufficient training to domains that improve more slowly, we propose Nash-MOPD, an adaptive domain scheduler based on a local minimax game using existing reverse-KL statistics. The equilibrium maximizes the smallest predicted proportional reduction under the local model and gives more prompts to domains that need more examples for comparable improvement. Consequently, Nash-MOPD responds to changing learning progress and supports domains that improve more slowly without extra teacher queries or rollouts. At step 100, Nash-MOPD exceeds both baselines’ final mathematics and code checkpoint scores on DeepSeek-R1-Distill-Qwen-1.5B and reaches Fixed’s on Qwen3-8B-Base. Furthermore, it leads eight of nine reported metrics and is best or tied best on all six mathematics and code accuracy metrics, respectively.
Source: Nash-MOPD: Optimizing Task Trade-offs in Multi-Teacher On-Policy Distillation