The Evaluation Crisis

The evaluation crisis is the gap between the success rates robot learning papers report and what those rates can establish. A success ratesuccess rateSuccess rate is the fraction of attempted episodes in which a policy completes the task, the standard headline metric of robot learning evaluation. A reported rate needs its trial count, time limit and scene distribution. Toyota Research Institute's Large Behavior Model study analyzed 1,800 real-world rollouts and warned of a significant risk of statistical noise from insufficient power. Snyder et al.'s STEP sequential test needs up to 32% fewer trials than the Lai and SAVI baselines. folds trial count, task length, scene setup and scorer into one number, and papers often report it with little or no information on the number of runs, initial conditions or success criteria^{Source 11}11 Kress-Gazit et al. 2024, Robot Learning as an Empirical Science: Best Practices for Policy Evaluation. Hadas Kress-Gazit, Kunimatsu Hashimoto, Naveen Kuppuswamy et al., 2024.

Paper figure: Pranav Atreya et al. / arXiv. Licence: CC BY 4.0.

What the success rate hides: the horizon

In an illustrative model, every decision succeeds with the same probability $p$ conditional on all earlier decisions succeeding, so a task of $n$ decisions completes with probability $p^n$^{Source 1}1 Meyerson et al. 2025, Solving a Million-Step LLM Task with Zero Errors. Elliot Meyerson, Giuseppe Paolo, Roberto Dailey et al., 2025. At $p = 0.95$, 30 decisions give 21.5% and 100 give 0.6%. Drag "How many steps the job has" to compare these examples.

Solving $p^n = q$ gives $p = q^{1/n}$, so an episode target of $q = 0.8$ needs 95.6% per step at $n = 5$ and 99.3% at $n = 30$. Behavior cloning compounds errors the same way: a policy with error rate $\varepsilon$ on the expert's states can make as many as $T^2\varepsilon$ mistakes over $T$ steps of its own^{Source 12}12 Ross et al. 2011, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. Stéphane Ross, Geoffrey J. Gordon, J. Andrew Bagnell, AISTATS 2011.

Benchmarks now measure long horizons directly. CALVIN scores 1,000 chains of five sequential tasks^{Source 13}13 Mees et al. 2021, CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks. Oier Mees, Lukas Hermann, Erick Rosete-Beas et al., 2021, and the 2026 BEHAVIOR Challenge ranks 100 full-length household tasks by average success with partial credit, with 20,000 teleoperated demonstrations totaling 1,950 hours^{Source 14}14 Lab 2026, 2026 BEHAVIOR Challenge. Stanford Vision and Learning Lab, BEHAVIOR project site, 2026.

Small samples limit comparison

Toyota Research Institute's Large Behavior Models extend the Diffusion Policydiffusion policyA diffusion policy is a policy generating a whole action sequence by conditional denoising diffusion instead of regressing one action. Introduced by Chi et al. in 2023, it represents multimodal demonstrations without averaging their modes. It executes only the first few steps of each sequence, then observes again and replans. recipe to multitask models trained on about 1,700 demonstration hours. The mixture holds 468 hours of real bimanual teleoperationteleoperationTeleoperation is direct remote control of a robot by a human operator, used in robot learning mostly to record expert demonstrations. In the cheap, precise ALOHA rig, the operator back-drives a leader arm twinning the follower joint for joint, so recordings are in the robot's own action space. Mobile ALOHA extended this leader-follower scheme to a mobile base for whole-body tasks., 45 simulation hours, 32 UMI hours and about 1,150 hours from a subset of Open X-Embodiment^{Source 3}3 Chi et al. 2023, Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. Cheng Chi, Zhenjia Xu, Siyuan Feng et al., 2023 ^{Source 2}2 TRI LBM Team et al. 2025, A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation. TRI LBM Team, Jose Barreiros, Andrew Beaulieu et al., 2025. TRI reports 1,800 real-world rollouts and over 47,000 simulation rollouts, with 50 real attempts per task per policy per condition, a target of 200 in simulation and missing simulation data for some combinations. Hardware comparisons were blind, with policy order randomized within bundles sharing an initial condition^{Source 2}2 TRI LBM Team et al. 2025, A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation. TRI LBM Team, Jose Barreiros, Andrew Beaulieu et al., 2025.

Simulation used automated predicates for scoring. TRI draws Bayesian posteriors over each policy's success probability under a uniform prior as violin plots, which avoid reading significance from overlapping confidence intervals; the posteriors describe single checkpoints and exclude variation from stochastic training. TRI also warns of a significant risk that many robotics papers measure statistical noise for lack of power^{Source 2}2 TRI LBM Team et al. 2025, A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation. TRI LBM Team, Jose Barreiros, Andrew Beaulieu et al., 2025. Snyder et al. give 10 or 50 trials as examples of small feasible samples, and a test that finds no significant difference does not show two policies are equivalent^{Source 4}4 Snyder et al. 2025, Is Your Imitation Learning Policy Better than Mine? Policy Comparison with Near-Optimal Stopping. David Snyder, Asher James Hancock, Apurva Badithela et al., RSS 2025.

Deep reinforcement learning met the same problem: without significance tests and standard reporting, claimed improvements are hard to judge^{Source 15}15 Henderson et al. 2018, Deep Reinforcement Learning that Matters. Peter Henderson, Riashat Islam, Philip Bachman et al., AAAI 2018, so interval estimates and performance profiles are advocated over point estimates^{Source 16}16 Agarwal et al. 2021, Deep Reinforcement Learning at the Edge of the Statistical Precipice. Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro et al., NeurIPS 2021.

STEP, a sequential test from Snyder et al., stops comparing two policies once enough evidence accumulates. It reports up to 32% fewer trials than the Lai and SAVI sequential baselines, assumes i.i.d. binary outcomes and uses a uniform risk-accumulation schedule in its experiments^{Source 4}4 Snyder et al. 2025, Is Your Imitation Learning Policy Better than Mine? Policy Comparison with Near-Optimal Stopping. David Snyder, Asher James Hancock, Apurva Badithela et al., RSS 2025. TRI uses the Lai test for binary comparisons, while STEP is a separate method, and Welch's t-test for task progress^{Source 2}2 TRI LBM Team et al. 2025, A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation. TRI LBM Team, Jose Barreiros, Andrew Beaulieu et al., 2025. A 2026 follow-up from Snyder and colleagues extends anytime-valid testing to partial credit and continuous metrics and reports up to 70% less evaluation effort than batch methods^{Source 17}17 Snyder et al. 2026, Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison. David Snyder, Apurva Badithela, Nikolai Matni et al., 2026.

Clean benchmarks overstate

LIBERO reports 130 tasks: ten each in LIBERO-Spatial, LIBERO-Object and LIBERO-Goal, plus LIBERO-100: 90 short-horizon pretraining tasks and ten long-horizon downstream tasks. Each task has 50 human-expert demonstrations collected by Spacemouse teleoperation^{Source 5}5 Liu et al. 2023, LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. Bo Liu, Yifeng Zhu, Chongkai Gao et al., 2023. LIBERO-Plus perturbs the benchmark along seven dimensions: object layout, camera viewpoint, robot initial state, language instruction, lighting, background texture and sensor noise. Models that score 95% on the clean benchmark fall below 30% under modest camera or initial-state shifts, and shuffling or removing the instruction barely moves performance, suggesting the models largely ignore the language^{Source 6}6 Fei et al. 2025, LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models. Senyu Fei, Siyin Wang, Junhao Shi et al., 2025.

Paper figure: Bo Liu et al. / arXiv. Licence: CC BY 4.0.

LIBERO-PRO reports that models above 90% on standard LIBERO collapse to 0.0% in its generalized setting^{Source 18}18 Zhou et al. 2025, LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization. Xueyang Zhou, Yangming Xu, Guiyao Tie et al., 2025, and THE COLOSSEUM measured 30 to 50% degradation per perturbation factor and at least 75% when perturbations combine^{Source 19}19 Pumacay et al. 2024, THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation. Wilbert Pumacay, Ishika Singh, Jiafei Duan et al., RSS 2024. On RoboDojo's 42 simulated and 18 real tasks, the best of 30 policies in the standard setting scored 21.98 and fell to 1.57 under randomization, a 92.9% relative drop^{Source 20}20 Chen et al. 2026, RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies. Tianxing Chen, Yue Chen, Zixuan Li et al., 2026. On RoboCasa365's household tasks, the best multi-task baseline, GR00T N1.5, averaged 20.0% success^{Source 21}21 Nasiriany et al. 2026, RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots. Soroush Nasiriany, Sepehr Nasiriany, Abhiram Maddukuri et al., ICLR 2026.

Newer benchmarks build the variation in. RoboTwin 2.0 randomizes clutter, lighting, background, tabletop height and language^{Source 22}22 Chen et al. 2025, RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation. Tianxing Chen, Zanxin Chen, Baijun Chen et al., 2025, and RoboLab's 120 tasks address the overlap between training and evaluation that trivializes success rates on existing benchmarks^{Source 23}23 Yang et al. 2026, RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies. Jenai Xuning Yang, Rishit Dagli, Alex Zook et al., Robotics: Science and Systems XXII (2026).

Simulation as an evaluation proxy

SIMPLER tests whether simulated performance predicts real performance^{Source 7}7 Li et al. 2024, Evaluating Real-World Robot Manipulation Policies in Simulation. Xuanlin Li, Kyle Hsu, Jiayuan Gu et al., 2024. It narrows the visual gap with real-image backgrounds and matched textures, and the control gap by fitting controller stiffness and damping through offline system identificationsystem identificationSystem identification is fitting a simulator's parameters to a real robot's measured dynamics, shrinking the sim-to-real gap by calibration. Hwangbo et al. found actuator error the dominant transfer obstacle on ANYmal and replaced its miscalibrated analytic model with a learned network from joint-command history to realized torque. It attacks the gap's dynamics component directly; domain randomization only averages over it.. The project site reports about 1,500 evaluation episodes each from real and simulated Google Robot and WidowX setups^{Source 24}24 Li et al. n.d., Evaluating Real-World Robot Manipulation Policies in Simulation (SIMPLER project page). Xuanlin Li, Kyle Hsu, Jiayuan Gu et al., simpler-env.github.io, n.d.; accessed 2026-10-13.

Agreement is scored with Pearson correlation, which can be unstable when policies perform similarly, and Mean Maximum Rank Violation, which weights wrong orderings by real performance margins. It varies by task: the WidowX carrot-placement comparison contains a ranking error. The experiments focus on rigid objects, and green-screening assumes fixed cameras and misses shadows^{Source 7}7 Li et al. 2024, Evaluating Real-World Robot Manipulation Policies in Simulation. Xuanlin Li, Kyle Hsu, Jiayuan Gu et al., 2024. Across 7 tasks and 5 policy architectures, SimFoundry's simulated evaluations reached a mean Pearson correlation of 0.911 with real results and a mean maximum rank violation of 0.018^{Source 25}25 Ranawaka et al. 2026, SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation. Nadun Ranawaka, Josiah Wong, Wei-Lin Pai et al., 2026.

Generative world models now act as evaluators (world-model evaluation). Google DeepMind validated a Veo world simulator against 1,600+ real-world evaluations of eight Gemini Robotics checkpoints^{Source 26}26 Gemini Robotics Team et al. 2025, Evaluating Gemini Robotics Policies in a Veo World Simulator. Gemini Robotics Team, Krzysztof Choromanski, Coline Devin et al., 2025, WorldEval reports a strong correlation with real-world performance^{Source 27}27 Li et al. 2025, WorldEval: World Model as Real-World Robot Policies Evaluator. Yaxuan Li, Yichen Zhu, Junjie Wen et al., 2025, and Ctrl-World, trained on DROID, ranks policies without real rollouts^{Source 28}28 Guo et al. 2025, Ctrl-World: A Controllable Generative World Model for Robot Manipulation. Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen et al., 2025. SureSim corrects simulator bias with a few paired real and simulated trials and saves over 20 to 25% of hardware evaluation effort for similar bounds^{Source 29}29 Badithela et al. 2025, Reliable and Scalable Robot Policy Evaluation with Imperfect Simulators. Apurva Badithela, David Snyder, Lihan Zha et al., 2025. Sim-to-Real Transfer covers simulation for training.

WorldOracle, from Haoyi Duan, Yuke Zhu and colleagues at AgiBot, NVIDIA and three universities (October 2026), feeds recorded actions from real robot episodes to seven action-conditioned world models and scores whether each predicted video matches what happened^{Source 34}34 Duan et al. 2026, WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions. Haoyi Duan, Liliang Chen, Nan Wang et al., 2026. On 100 two-minute AgiBot G2 episodes, DreamDojo's single-view score out of 100 fell from 50.56 over the first 30 seconds to 27.08 over the full episode, and Cosmos 3 Nano's from 53.99 to 26.60^{Source 34}34 Duan et al. 2026, WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions. Haoyi Duan, Liliang Chen, Nan Wang et al., 2026. AgiBot's own GE-Sim 2.0, which shares four authors with the benchmark, fell from 73.86 to 63.33^{Source 34}34 Duan et al. 2026, WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions. Haoyi Duan, Liliang Chen, Nan Wang et al., 2026 ^{Source 35}35 Qiu et al. 2026, GE-Sim 2.0: A Roadmap Towards Comprehensive Closed-loop Video World Simulators for Robotic Manipulation. Boxiang Qiu, Liliang Chen, Yue Liao et al., 2026.

Blind, distributed, pairwise

RoboArena distributes evaluation across seven universities on the DROID Franka Panda platform^{Source 10}10 Khazatsky et al. 2024, DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. Alexander Khazatsky, Karl Pertsch, Suraj Nair, 2024 ^{Source 8}8 Atreya et al. 2025, RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies. Pranav Atreya, Karl Pertsch, Tony Lee et al., 2025. Evaluators receive anonymous policy endpoints, choose tasks and scenes, and run each pair double-blind and back-to-back from closely matched initial conditions, recording progress scores, preferences and explanations. Its study reports 612 pairwise comparisons across seven DROID-trained policies, and against a 4,284-rollout oracle its rankings agreed better than a conventional evaluation of 17 tasks and 44 episodes per policy^{Source 8}8 Atreya et al. 2025, RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies. Pranav Atreya, Karl Pertsch, Tony Lee et al., 2025. Chatbot Arena ranks language models the same pairwise, crowdsourced way, from over 240K votes^{Source 30}30 Chiang et al. 2024, Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng et al., 2024.

RoboChallenge hosts robots at its own site and exposes asynchronous observation and action APIs; model inference stays with the submitting user. Its first benchmark, Table30, holds tabletop tasks; held-out demonstration frames guide scene resets, but lighting and camera drift remain, and the system cannot verify the remote model matches its submitted name^{Source 9}9 Yakefu et al. 2025, RoboChallenge: Large-scale Real-robot Evaluation of Embodied Policies. Adina Yakefu, Bin Xie, Chongyang Xu et al., 2025.

OpenVLA's evaluation needed over 2,500 rollouts and over 100 hours of human labour, and AutoEval's automatic resets and success detection let evaluation run around the clock^{Source 31}31 Zhou et al. 2025, AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real World. Zhiyuan Zhou, Pranav Atreya, You Liang Tan et al., 2025. XPolicyLab cuts the integrations needed to connect $N$ policies to $M$ environments from $O(NM)$ to $O(N+M)$^{Source 32}32 Community et al. 2026, XPolicyLab: A Unified Standard and Open Ecosystem for Robot Policy Evaluation and Deployment. XPolicyLab Community, Tianxing Chen, Yue Chen et al., 2026.

Reading a result table

• How many rollouts produced the number, and is its uncertainty reported^{Source 2}2 TRI LBM Team et al. 2025, A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation. TRI LBM Team, Jose Barreiros, Andrew Beaulieu et al., 2025?
• How long is the task?
• What survives perturbation^{Source 6}6 Fei et al. 2025, LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models. Senyu Fei, Siyin Wang, Junhao Shi et al., 2025?
• Who scored the rollouts, with what agreement against the tested population^{Source 8}8 Atreya et al. 2025, RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies. Pranav Atreya, Karl Pertsch, Tony Lee et al., 2025?
• What correlation backs a simulation number^{Source 7}7 Li et al. 2024, Evaluating Real-World Robot Manipulation Policies in Simulation. Xuanlin Li, Kyle Hsu, Jiayuan Gu et al., 2024?

StopWatch, from Xinhao Liu, Pieter Abbeel, Guanya Shi and colleagues at Amazon FAR, NYU, UC Berkeley and Carnegie Mellon (October 2026), checked the labels that score HM3D-OVON, an object-goal navigation benchmark. A stop counts only when the object's annotated name matches the goal word, so a robot that stops at a couch annotated as sofa fails when asked for a couch, and a manual audit dropped 23% of the unseen-category validation episodes for missing or wrong labels. The frontier-exploration expert that generated the training demonstrations succeeds in 72% of episodes under the official scoring and 96% under the authors' corrected scoring of the remaining episodes, and their system's lead over the strongest baseline, LightNav-0, grows from 6.4 to 8.8 points^{Source 36}36 Liu et al. 2026, StopWatch: Goal Grounding for Object-goal Navigation. Xinhao Liu, Yunsheng Tian, Yuheng Qiu et al., 2026.

Eval-Actions argues binary success rates hide process-level differences and provides 13K+ real-robot episodes over 150+ tasks^{Source 33}33 Liu et al. 2026, Eval-Actions: Fine-Grained Execution Quality Evaluation for Robotic Manipulation. Mengyuan Liu, Juyi Sheng, Peiming Li et al., 2026, and fine-grained task progress separates policies with fewer trials than binary success^{Source 17}17 Snyder et al. 2026, Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison. David Snyder, Apurva Badithela, Nikolai Matni et al., 2026. The reliability gap follows the same arithmetic into deployment.