AI Insight
This study challenges the assumption that Q-functions (value estimators in reinforcement learning) need to be pretrained on offline data before fine-tuning a pretrained policy. The researchers found that randomly initialized Q-functions perform similarly to pretrained ones during online fine-tuning, because pretrained Q-functions target the original policy rather than the policy that emerges during fine-tuning. They propose a new initialization method called IPE that trains multiple diverse policies and pools their experiences, achieving 1.26x better performance than traditional Q-function pretraining across continuous control tasks.
Why it matters
This finding could simplify and accelerate the development of reinforcement learning systems by eliminating unnecessary pretraining steps for Q-functions. The proposed IPE method offers a practical alternative that improves fine-tuning efficiency, potentially reducing computational costs and development time for AI systems that learn through trial and error.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) this raises a natural question: given a pretrained policy, should the Q-function be pretrained on offline data too? Conventional wisdom suggests it should, but recent results show that online RL with a randomly-initialized Q-function can result in highly performant and reliable policies without needing to pretrain the Q-function. In this paper, we systematically study whether pretraining the Q-function actually helps when fine-tuning on top of a pretrained base policy. We find, surprisingly, that naive Q-function pretraining often provides little benefit over random initialization. We show this stems from a fundamental mismatch: the Q-function learned during pretraining targets the pretrained policy’s Q-function, not the Q-function that online fine-tuning converges to, and this gap persists even after offline value maximization. Motivated by this finding, we propose Initialization via Policy Ensemble (IPE), a simple method that trains multiple diverse policies and uses their pooled rollouts to bootstrap the Q-function learning in online RL. Across a suite of challenging continuous control benchmarks, IPE yields an average 1.26x improvement in fine-tuning performance over naive Q-function pre-training.
Source: Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?