The Reliability Gap

A robot that finishes a task 80% of the time is a demo. A robot that finishes it 99.9% of the time is a product. The distance between those numbers is the reliability gap: the difference between the success ratessuccess rateThe standard headline metric of robot learning evaluation: the fraction of attempted episodes in which the policy completes the task. Trial count, time limit and scene distribution belong with the reported rate. Toyota Research Institute's Large Behavior Model study analyzed 1,800 real-world rollouts and warned of a significant risk of statistical noise from insufficient power, not a measured prevalence of incorrect papers. An inconclusive comparison does not establish policy equivalence. a system demonstrates in controlled settings and the rates that deployment requires. Bessemer's 2026 robotics analysis states the framing directly: getting from 80% task success to 99.9% is not a linear problem, and the last stretch requires fundamentally different approaches than the first Levine 2026↓Bessemer Predicts: Robotics and physical AIJeremy Levine, Talia Goldberg, Janelle Teng Wade et al., 2026. This module covers why the gap is structural, what is closing it, and what the deployment record shows once claims are separated from verified numbers.

The arithmetic of compounding

A manipulation task is a chain of decisions, and the chain succeeds only when every link does. If each step succeeds independently with probability $p$, an $N$-step task completes with probability $p^N$. The exponent is the whole story. A policy with 95% per-step success completes a 30-step task 21% of the time. At 99% per-step the same task completes 74% of the time. Even at 99.9% per-step, a figure almost no system publishes, a 30-step task completes 97% of the time and still fails once in thirty runs. The per-step number and the task number are different metrics, and a result reported without its horizon cannot be compared against either. Drag the per-step success slider below from 95% up toward 100 and watch how late the episode number recovers.

The calculator computes the product at any setting, and its sliders span the full range so the boundary cases stay honest: at 100% per-step the task always completes, at 0% it never does, and at a horizon of one step the readout equals the per-step value. Two refinements matter for reading real systems. First, the independence assumption is generous: a mistake carries the robot into a state outside its training distribution, where the next mistake is more likely, so compounding errorcompounding errorThe accumulation of small per-step mistakes over the course of a sequential task: each error carries the robot into a state slightly outside its training data, where the next error is more likely, so total deviation grows faster than the per-step error rate. Ross, Gordon, and Bagnell bounded the effect for imitation learning with total cost quadratic in episode length, and action chunking attacks the same effect from the other side by shrinking the number of sequential decisions in an episode. makes the true curve worse than the calculator's. Second, the inverse reading sets the engineering bar. Holding 99% end-to-end success on a 50-step task requires about 99.98% per-step reliability, a figure no generalist policy publishes.

Why the last stretch is different

Moving per-step reliability from 95% to 99% is an engineering problem with known methods: more data, better models, targeted evaluation. Moving from 99% to 99.9% is a different kind of problem, because the remaining failures concentrate in the long tail: unusual object configurations, lighting, and contact dynamics that any finite training set underrepresents Levine 2026↓Bessemer Predicts: Robotics and physical AIJeremy Levine, Talia Goldberg, Janelle Teng Wade et al., 2026. Lisa Yan, co-founder and CEO of Argus Systems and previously at Waymo, describes the deployment experience in Bessemer's analysis: closing the gap between 99% and 99.9% reliability is "a steep hill climb that takes longer than most people realize" Levine 2026↓Bessemer Predicts: Robotics and physical AIJeremy Levine, Talia Goldberg, Janelle Teng Wade et al., 2026. Bessemer's own forecast is more optimistic about timing, arguing the field's ChatGPT moment is not years away, and the disagreement is about the slope of the improvement curve rather than the existence of the gap Levine 2026↓Bessemer Predicts: Robotics and physical AIJeremy Levine, Talia Goldberg, Janelle Teng Wade et al., 2026.

The classical safety toolbox does not close the gap either. Formal methods such as barrier functions and reachability analysis do not scale to high-dimensional manipulation with learned policies, and as of mid-2026 no deployed vision-language-action modelvision-language-action modelA robot policy built by adapting a vision-language model, pretrained on web-scale image and text data, to output robot actions. RT-2 established the approach by representing actions as text tokens and co-fine-tuning on web and robot data together, showing that semantic knowledge from web pretraining transfers into control. The same recipe underlies OpenVLA and the π series of generalist policies. publishes formal safety guarantees. Assurance in practice rests on empirical evaluation and runtime monitoring, which makes the reliability gap and the evaluation crisis the same problem seen at different altitudes: one is about making failure rare, the other about being able to measure how rare it is (see The Evaluation Crisis).

What is closing the gap

Three methods are doing measurable work, each with a known ceiling.

RL from experience. Physical Intelligence's π*0.6 with Recap trains on the robot's own deployment experience, cutting failure rates by 2x or more on hard tasks and reaching above 90% success on espresso, laundry, and box-assembly after on-robot RL, with more than 2x throughput on some of the hardest tasks Amin 2025↓π*0.6: a VLA that Learns from ExperienceAli Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025. RL-100 goes further on its task set: 100% success across 1,000 episodes on 8 tasks, including 250 consecutive successes on one task, and a juicing robot that served customers in a shopping mall for about seven hours without failure Lei 2025↓RL-100: Performant Robotic Manipulation with Real-World Reinforcement LearningKun Lei, Huanyu Li, Dongjie Yu et al., 2025. The ceiling is that both are per-task specialist results earned through on-robot training, not generalist reliability. RL-100's own zero-shot figure under environmental and dynamics shifts is about 90%, which is the gap reasserting itself one distribution over Lei 2025↓RL-100: Performant Robotic Manipulation with Real-World Reinforcement LearningKun Lei, Huanyu Li, Dongjie Yu et al., 2025.

Parada 2026↓Gemini Robotics 2 brings whole body intelligence to robotsCarolina Parada, 2026 Runtime monitors. Google's ASIMOV-Agentic benchmark evaluates the orchestration layer around the policy rather than the policy itself: whether an embodied reasoning agent refuses unsafe tool calls from the VLA, predicts whether a task is feasible, and requests human intervention when uncertain Google DeepMind 2026↓Asimov Agentic Safety EvaluationGoogle DeepMind, 2026.

Parada 2026↓Gemini Robotics 2 brings whole body intelligence to robotsCarolina Parada, 2026 DeepMind describes Gemini Robotics ER 2 as its safest robotics model to date on safety-constraint-following and human-proximity benchmarks, with the ability to detect nearby humans and trigger safe stops. Monitoring catches failures instead of preventing them, so it raises effective reliability without changing the policy's own success rate.

Human in the loop. π0.7 accepts step-by-step language coaching that steers it through tasks it has not seen, recovering from failures that would otherwise end the episode Ai 2026↓π0.7: a Steerable Model with Emergent CapabilitiesBo Ai, Ali Amin, Raichelle Aniceto et al., 2026. Supervised operation bridges the gap for paying customers while the autonomy catches up, but a bridged gap is not a closed one: the reliability number that matters is the one with the coach removed.

What is actually deployed

The gap is easiest to see in the deployment record, where claimed and verified numbers diverge. Circulating figures put Tesla's cumulative Optimus builds in the tens of thousands and Figure's deployments above ten thousand; neither figure comes from the companies they describe, and Tesla has never published a production count Noreika 2026↓Humanoid Robots in 2026: What Is Actually DeployedAlius Noreika, 2026. The verified record is narrower. Agility's Digit has accumulated more than 65,000 operating hours across nine customer facilities. Figure's eleven-month pilot at BMW Spartanburg logged more than 1,250 hours on a live assembly line. Unitree shipped roughly 5,500 humanoid units in 2025, with its G1 starting near $16,000, though volume has not produced profit: Q1 2026 adjusted net profit fell 52.55% year over year Noreika 2026↓Humanoid Robots in 2026: What Is Actually DeployedAlius Noreika, 2026.

The dashboard keeps the two classes apart. A verified row is documented against company statements, filings, or named customers; a claimed row is a circulating figure without a company source, or a vendor-run demonstration without an independent audit. Figure's eight-hour autonomous shift of May 2026 sits in the claimed rows deliberately: the livestream was real, but a single task at a single site with no published success rate is a demonstration, not a deployment metric Belmonte 2026↓Figure AI's Helix-02 Robots Complete Full 8-Hour Autonomous Shifts as Humanoid Race IntensifiesKyle Belmonte, 2026. The valuation context belongs here as well. Figure's private valuation of $39B (September 2025) exceeds Goldman Sachs' projection for the entire humanoid market in 2035, $38B, a distance that only closes if reliability does Noreika 2026↓Humanoid Robots in 2026: What Is Actually DeployedAlius Noreika, 2026.

What solved would look like

A closed reliability gap has a measurable shape: a deployed system with a documented mean time between failures above 1,000 hours on real tasks, formal safety guarantees for human-robot interaction, and a published protocol for the failure modes that remain. No current system publishes any of the three. Until one does, the honest unit of progress is not the demo but the verified operating hour, and on that metric the field measures tens of thousands of hours against the millions that general-purpose deployment implies.

The second of those three is the one with an established answer everywhere except robot learning. Safety and assurance covers the standards stack industrial robotics already has, why a learned policy cannot be assigned a rating under any of it, and the wrapper architecture teams ship in place of the guarantee.