AI Insight
AcceRL is a new distributed asynchronous reinforcement learning framework designed to train large-scale Vision-Language-Action models more efficiently by separating environment simulation, model inference, and training updates into independent processes. This approach eliminates synchronization bottlenecks that plague traditional systems, achieving 2.4 times faster processing throughput compared to existing synchronous methods. When integrated with pre-trained world models, the framework demonstrates up to 200 times improvement in sample efficiency on robotic manipulation tasks from the LIBERO benchmark suite.
Why it matters
This framework addresses two critical bottlenecks in training embodied AI systems: computational efficiency and the high cost of collecting real-world or simulated training data. By dramatically reducing both training time and the number of environment interactions needed, AcceRL could accelerate the development of more capable robots for manufacturing, healthcare, and domestic assistance applications.
Understand the Science
arXiv:2603.18464v3 Announce Type: replace
Abstract: Reinforcement learning (RL) for large-scale Vision-Language-Action (VLA) models is severely bottlenecked by synchronization barriers and the high cost of environment data acquisition. To overcome these challenges, we propose AcceRL, a distributed asynchronous RL framework that physically isolates environment rollouts, model inference, and gradient updates. By eliminating the cascading long-tail idle bubbles inherent in synchronous systems, AcceRL maximizes hardware utilization and ensures scalable throughput. Furthermore, AcceRL features a modular design that supports the integration of diverse, plug-and-play world models into its distributed pipeline. Extensive experiments demonstrate that the base framework achieves highly competitive performance across all four LIBERO~cite{liu2023libero} task suites. Systematically, the asynchronous architecture delivers a $2.4times$ throughput speedup over leading synchronous baselines. Algorithmically, by leveraging a world model pre-trained on 1,000 offline trajectories, AcceRL achieves up to a $200times$ improvement in online sample efficiency on LIBERO-Spatial, establishing a robust framework that is both sample-efficient and time-efficient for embodied AI. Code is included in the supplementary material. Code is available at https://github.com/distanceLu/AcceRL.