AI Insight
This paper introduces QGF (Q-Guided Flow), a reinforcement learning algorithm that performs policy optimization entirely at test time rather than during training. The method trains a flow-based policy through standard behavioral cloning and a separate value function, then uses value gradients at test time to guide the policy toward higher-value actions without additional policy learning. QGF demonstrates superior performance compared to other test-time RL methods on offline RL benchmarks with high-dimensional action spaces and achieves competitive results with state-of-the-art training-time algorithms while being computationally cheaper.
Why it matters
This approach addresses major stability and scalability issues in incorporating expressive policies like diffusion and flow models into reinforcement learning for robot control. By separating policy improvement from training, it offers a more stable and cost-effective alternative for real-world robotic applications while maintaining performance quality.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Expressive continuous control policies, such as diffusion and flow models, form the backbone of recent advances in scaling imitation learning for simulated and real robot control. While they are known to scale stably in the supervised imitation learning setting, incorporating them into reinforcement learning (RL) pipelines for policy improvement has proven more difficult. It often requires specialized training objectives or backpropagating through denoising processes, which cause well-known issues with stability and affect scalability. In this paper we study the question of whether simple policy improvement schemes at test time alone, leaving stable supervised policy training intact, can be a competitive alternative which sidesteps these issues. To this end, we propose QGF (Q-Guided Flow), an RL algorithm that performs policy optimization entirely at test time. QGF works by pre-training both a reference flow policy (via a standard behavioral cloning objective) and a value function critic and, at test time, using the value gradient to guide the reference policy to generate higher-value actions without any additional policy learning. Empirically, QGF outperforms prior test-time RL methods on single-task and goal-conditioned offline RL benchmarks with high-dimensional action spaces, and is competitive with state-of-the-art training-time algorithms while being much cheaper to run. Moreover, it exhibits favorable scaling with model size by avoiding the instability of actor-critic training, offering a practical and effective alternative RL algorithm with expressive policies.
Source: Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning