AI Insight
EfficientTDMPC is a new model-based reinforcement learning algorithm designed for continuous control tasks that achieves improved sample efficiency through three key innovations: an aggregate multi-horizon planning objective that averages value estimates across different prediction depths, ensemble methods for state-action value estimation, and pessimistic reanalyze which penalizes uncertain return estimates during policy training. The method demonstrates state-of-the-art sample efficiency on benchmark tests including HumanoidBench and the DeepMind Control Suite, building upon the TD-MPC algorithm family by addressing sources of error in both learned models and value networks.
Why it matters
Sample-efficient reinforcement learning is critical for real-world applications where data collection is expensive or time-consuming, such as robotics and autonomous systems. By reducing the number of interactions needed to learn effective control policies, this approach could make advanced AI control systems more practical and accessible for deployment in physical systems.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: We introduce EfficientTDMPC, a sample-efficient model-based reinforcement learning method for continuous control built on the TD-MPC family of algorithms. Central to this family is a planner that aims to find an action sequence that maximizes the estimated return. The return is estimated using a learned model and value networks, each of which can introduce error. EfficientTDMPC introduces three contributions that improve performance by aiming to reduce this error. First, we introduce an aggregate multi-horizon planning objective that evaluates the value at different rollout depths and averages them. Second, we introduce ensembles for state-action value estimation to value-equivalent/MuZero-style model-based RL methods. Third, we add pessimistic reanalyze, which penalizes uncertain return estimates when creating policy targets. We evaluate EfficientTDMPC on HumanoidBench and the DeepMind Control Suite, to the best of our knowledge, it is the new state of the art on both domains in terms of sample efficiency.
Source: EfficientTDMPC: Improved MPC Objectives for Sample-Efficient Continuous Control