AI Insight
This study presents a reinforcement learning approach for training quadrupedal robots to walk more efficiently by combining traditional policy gradient methods (PPO) with model-based synthetic data generation in a Dyna-style framework. The researchers developed a learned transition model that generates short synthetic trajectory extensions, gradually integrated into training through a scheduled approach that avoids using the model when prediction accuracy is initially low. Testing on simulated Unitree Go1, Go2, and ANYmal robots demonstrated that the method achieves comparable locomotion performance while reducing required simulation steps by approximately 28.7% and training time by 12.24%.
Why it matters
This work addresses a critical bottleneck in robot learning by significantly reducing the computational resources and time needed to train quadrupedal locomotion controllers. The improved sample efficiency could accelerate the development and deployment of legged robots for real-world applications including search and rescue, industrial inspection, and autonomous navigation in complex environments.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
-cross
Abstract: Traditional on-policy reinforcement learning (RL) controllers for quadrupedal locomotion often suffer from low data efficiency, requiring millions of interactions with simulated environments to achieve stable control. We integrate model-based techniques that improve sample efficiency by augmenting PPO rollouts with synthetic data in a Dyna-style framework. Our method employs a learned transition model to generate short-horizon synthetic tails for each trajectory, anchored by physics-based simulation to preserve stability. A predefined scheduling strategy gradually integrates synthetic transitions, preventing model usage during early training stages when prediction accuracy is low. Through extensive ablation studies, we analyze how varying data parameters influence PPO’s learning behavior. Finally, we validate our method in simulation on a Unitree Go1 robot, reaching convergence with substantially fewer simulation steps (19.64M vs. 27.53M) and a 12.24% reduction in wall-clock training time, without compromising policy performance or convergence. Cross-platform experiments on ANYmal and Unitree Go2 further confirm the framework’s ability to learn high-dimensional locomotion control with substantially reduced simulation experience, despite reward trade-offs on complex morphologies.
Source: Learning to Walk With Less: A Dyna-Style Approach to Quadrupedal Locomotion