AI Insight
This paper introduces DPRM, a plug-in module that improves diffusion language models by optimizing the order in which tokens are generated during text synthesis. Unlike existing methods that use random masking or simple confidence-based ordering, DPRM employs a process-reward-guided approach that transitions from confidence-driven to reward-optimized token ordering through online learning. The authors provide theoretical convergence guarantees and demonstrate improvements across nine different tasks spanning language reasoning, protein generation, DNA sequences, and multimodal applications.
Why it matters
This work addresses a fundamental challenge in non-autoregressive language generation by providing a theoretically grounded and empirically validated method for token ordering that can be integrated into existing diffusion models without architectural changes. The approach has potential applications across diverse domains including computational biology, drug discovery, and multimodal AI systems where generation quality is critical.
arXiv:2604.24357v2 Announce Type: replace
Abstract: Diffusion language models generate without a fixed left-to-right order, leaving token ordering as a central algorithmic choice. Existing systems mainly use random masking or confidence-driven ordering, which respectively suffer from train–test mismatch and myopic exploration. We introduce DPRM (Doob -transform Process Reward Model), a plug-in token-ordering module that keeps the host architecture, denoising objective and supervision unchanged, and modifies only the ordering policy. DPRM starts from confidence-driven ordering and gradually shifts to process-reward-guided ordering through online estimates. We characterize the exact DPRM policy as a reward-tilted Gibbs reveal law, prove convergence of its stagewise Soft-BoN approximation, show that the online bucketized controller tracks the exact DPRM score at empirical-Bernstein rates, and establish a sample-complexity advantage under tractable optimization assumptions.
Across nine hosts covering language reasoning, test-time scaling, protein, single-cell, molecular, DNA, text-to-image generation, and VQA, DPRM order variants improve several language, DNA, and multimodal settings while also identifying boundary cases where confidence-only ordering or task-specific utilities are preferable. Code is available at: https://github.com/DakeBU/DPRM-DLLM
Source: DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Diffusion Language Models