AI Insight
RoboDojo is a new benchmark system designed to comprehensively evaluate generalist robot manipulation policies across both simulated and real-world environments. The benchmark includes 42 simulation tasks testing five key capabilities (generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following) and 18 real-world tasks that expose policies to actual physical deployment challenges. The researchers integrated 30 existing policies into their XPolicyLab framework and established a public leaderboard with systematic performance analysis.
Why it matters
This unified benchmark addresses a critical gap in robotics research by enabling standardized, reproducible evaluation of robot manipulation policies that bridges the simulation-to-reality divide. The system's remote cloud access and standardized evaluation protocols make comprehensive robot policy testing more accessible and scalable, potentially accelerating the development of more capable and reliable robotic systems for real-world applications.
Understand the Science
arXiv:2607.04434v3 Announce Type: replace-cross
Abstract: Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remain limited in systematically evaluating their capabilities. Many rely on simple, short-horizon, or skill-narrow tasks with limited capability coverage, and are often conducted only in simulation or only in the real world. Simulation enables scalable feedback but misses physical deployment challenges, while real-world evaluation is costly, time-consuming, and difficult to reproduce. We introduce RoboDojo, a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies. RoboDojo includes 42 simulation tasks and 18 real-world tasks covering diverse and complementary manipulation capabilities. The simulation benchmark evaluates five dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following, while the real-world benchmark exposes policies to challenging physical-world deployment conditions. RoboDojo supports scalable evaluation through heterogeneous parallel simulation in Isaac Sim and provides RoboDojo-RealEval, a reproducible real-world evaluation system with remote cloud access, standardized hardware, scene reset, evaluation protocol, and deployment interface. Together with XPolicyLab, policies can be integrated once and evaluated across simulation and real-world settings with minimal adaptation. We integrate 30 policies into XPolicyLab and evaluate them on RoboDojo, establishing a public leaderboard and systematic analysis of current policy performance. The website is available at http://robodojo-benchmark.com/.