The Data Bottleneck

Language model pretraining runs on text scraped at internet scale. GPT-3 consumed 300 billion tokens in 2020 Brown 2020↓Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, NeurIPS 2020; Llama 3 consumed over 15 trillion in 2024 Meta AI 2024↓Introducing Meta Llama 3: The most capable openly available LLM to dateMeta AI, 2024, a scale the open FineWeb corpus replicates from 96 Common Crawl snapshots Penedo 2024↓The FineWeb Datasets: Decanting the Web for the Finest Text Data at ScaleGuilherme Penedo, Hynek Kydlíček, Loubna Ben allal et al., NeurIPS 2024. Robot learning has no equivalent. Open X-EmbodimentOpen X-EmbodimentA 2023 collaboration across 34 labs that pooled 60 existing robot datasets into one standardized format: over a million trajectories spanning 22 robot embodiments and 527 skills. The project also trained the RT-X models on the pool, and found that a policy trained across many robots outperforms versions trained on each robot's data alone, with skills carrying over to embodiments the policy saw little of. pools over a million trajectories across 22 robot embodiments Open X-Embodiment Collaboration 2023↓Open X-Embodiment: Robotic Learning Datasets and RT-X ModelsOpen X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman et al., 2023. Its total duration is unknown in the inspected paper and project-page text; trajectory count alone does not establish hours. The largest single-organization release, AgiBot World, holds 1,001,552 trajectories, and it does publish an hour count: 2,976 hours, which works out to about 11 seconds per trajectory AgiBot-World-Contributors 2025↓AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied SystemsAgiBot-World-Contributors, Qingwen Bu, Jisong Cai et al., 2025. A million trajectories, in other words, is under three thousand hours. DROID reports 76,000 successful trajectories totaling 350 interaction hours. Fifty data collectors used 18 robots across 13 institutions over 12 months Khazatsky 2024↓DROID: A Large-Scale In-The-Wild Robot Manipulation DatasetAlexander Khazatsky, Karl Pertsch, Suraj Nair, 2024. The paper also reports roughly 16,000 unsuccessful trajectories, released separately from the headline count. The 12 months describe elapsed collection time, not a full working year for each collector or robot. Toyota Research Institute's Large Behavior Model program trained on about 1,700 hours in total, spread across its own bimanual data, simulation, UMI, and Open X-Embodiment TRI LBM Team 2025↓A Careful Examination of Large Behavior Models for Multitask Dexterous ManipulationTRI LBM Team, Jose Barreiros, Andrew Beaulieu et al., 2025. That gap, and everything the field is doing about it, is the subject of this module.

The chart below separates dataset counts from a teaching projection. Drag the teleoperation rigs slider to see how fleet size changes collection time under two authored hypothetical rates. The 10,000-hour and 1,000,000-hour targets are hypothetical too, not OXE duration or measured frontier requirements.

Two data universes, nine orders of magnitude apart

Text is scraped. The internet accumulates language as a byproduct of everything people publish, so assembling a 15-trillion-token corpus is a filtering problem, not a collection problem. Real-world robot data is different. Every hour of it is produced by a physical machine interacting with the world in real time, usually driven by a paid human teleoperator. There is no crawl for contact.

The chart at the top of this module plots both universes. Robot and human demonstration data sits on the bottom lane, read off the hours axis: DROID at 350 hours, TRI's LBM corpus near 1,700, AgiBot World at 2,976, EgoScale's human video at 20,854. Open X-Embodiment remains listed as unknown duration and is not plotted or ranked by hours. Language corpora sit on the left lane, read off the tokens axis: GPT-3 at 300 billion, Llama 3 at 15 trillion. The dashed diagonal spans the empty plane between the two largest entries. No exchange rate between an hour of robot data and a token of text is drawn, because no honest one exists.

The slider projects a teleoperationteleoperationDirect remote control of a robot by a human operator, used in robot learning mostly to record expert demonstrations. The ALOHA rig made the setup cheap and precise: the operator back-drives a leader arm that is a joint-for-joint twin of the follower the policy will run on, so the recorded motion is already in the robot's own action space. Mobile ALOHA extended the same leader-follower scheme to a mobile base for whole-body tasks. farm using authored hypothetical inputs. The dedicated-farm rate assumes 1,000 productive hours per rig-year, equivalent to four hours on each of 250 working days. The low-rate scenario assumes seven. DROID's 350 hours divided by 50 collectors happens to equal seven, but that quotient is not a measured annual rate per rig: the source does not report each collector's or robot's working exposure. Neither scenario establishes how fast a real farm would collect data. Ask two questions of a dataset claim: how many hours, because episode counts hide trajectory length, and how diverse, which is where the scaling lawsscaling lawAn empirical regularity between a system's performance and the resources it consumes, parameters, data, or compute, first made precise for language models and now measured for robot learning. Lin and colleagues fit imitation-learning success to the training data and found that generalization to new objects and environments tracks data diversity rather than raw hours. EgoScale extended the measurement to egocentric human video, reporting a log-linear relationship between hours of human data and dexterous-manipulation success across four orders of magnitude. come in.

What the scaling laws say: diversity beats density

The obvious answer to the bottleneck is to collect more. Two papers now give quantitative guidance on what "more" should mean, and both point away from raw volume.

Lin et al. collected over 40,000 demonstrations and executed more than 15,000 real-world rollouts to map how generalization scales in imitation learningimitation learningLearning a policy from expert demonstrations rather than from a reward signal: the expert's recorded state-action pairs become a supervised training set, and the fitted mapping from observed state to action is the policy. Pomerleau's ALVINN (1988) instead trained on simulated road images before steering NAVLAB, and the recipe still underlies most learned manipulation. Its structural weakness is that the training distribution comes from the expert while deployment visits the states the learner itself induces, the mismatch DAgger was designed to repair. Lin 2024↓Data Scaling Laws in Imitation Learning for Robotic ManipulationFanqi Lin, Yingdong Hu, Pingyue Sheng et al., ICLR 2025, 2024. Success on unseen environments and objects follows a rough power law in the number of training environments and objects. Demonstrations per environment saturate quickly: past about 50 per environment, adding more yields little, while adding environments keeps paying. Their efficient recipe, 32 environments with 50 demonstrations each, reached about 90% success in novel environments with unseen objects, and the data for it was collected by four operators in a single afternoon.

Shi et al., working on the AgiBot World stack, took diversity apart into three axes: task, embodiment, and expert Shi 2025↓Is Diversity All You Need for Scalable Robotic Manipulation?Modi Shi, Li Chen, Jin Chen et al., 2025. Task diversity mattered more than per-task demonstration count, echoing Lin et al. Multi-embodiment pretraining turned out to be optional: models pretrained on high-quality single-embodiment data transferred to other platforms and scaled better during fine-tuning than multi-embodiment models. Expert diversity actively hurt. Operators with different styles produce multimodal action distributions, velocity multimodality in particular, that confound imitation. Debiasing those distributions produced GO-1-Pro, a 15% performance gain the authors equate to a 2.5x increase in pretraining data.

The strategic reading is that the bottleneck is breadth, not hours. An hour of teleoperation in a new kitchen with new objects is worth more than an hour in a scene the policy has already seen fifty times. A data budget spent maximizing scene and task coverage beats the same budget spent deepening coverage of familiar setups, and a fleet of operators with identical style can be worth more than a larger fleet with mixed styles, unless the mixed styles are debiased.

Around the bottleneck: data that does not need a robot

If breadth is the constraint, sources that are cheap and broad beat sources that are expensive and narrow, even when they lack robot actions.

UMI removes the robot from collection entirely: an operator holds a 3D-printed gripper fitted with a GoPro and performs the task wherever the task naturally lives, and policies trained on the handheld data deploy zero-shot onto real arms Chi 2024↓Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild RobotsCheng Chi, Zhenjia Xu, Chuer Pan et al., 2024. Apple's EgoDex used the Vision Pro's hand tracking to record 829 hours of dexterous human manipulation with per-joint 3D poses across 194 tabletop tasks, again with no robot in the loop Hoque 2025↓EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric VideoRyan Hoque, Peide Huang, David J. Yoon et al., ICLR 2026, 2025. Ego4D offers 3,670 hours of egocentric daily-life video from 931 camera wearers across 74 locations, valuable for visual pretraining even though it carries no action labels Grauman 2022↓Ego4D: Around the World in 3,000 Hours of Egocentric VideoKristen Grauman, Andrew Westbury, Eugene Byrne, CVPR 2022.

EgoScale is the strongest evidence so far that this route scales. Zheng et al. trained a vision-language-action model on 20,854 hours of action-labeled egocentric human video, more than twenty times larger than prior efforts, and uncovered a log-linear scaling law between human data scale and validation loss Zheng 2026↓EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human DataRuijie Zheng, Dantong Niu, Yuqi Xie et al., 2026. That validation loss, in turn, correlated with real-robot performance, which makes human video a predictable supervision source rather than a hopeful one. Their two-stage recipe, large-scale human pretraining followed by a lightweight aligned human-robot mid-training, improved average success ratesuccess rateThe standard headline metric of robot learning evaluation: the fraction of attempted episodes in which the policy completes the task. Trial count, time limit and scene distribution belong with the reported rate. Toyota Research Institute's Large Behavior Model study analyzed 1,800 real-world rollouts and warned of a significant risk of statistical noise from insufficient power, not a measured prevalence of incorrect papers. An inconclusive comparison does not establish policy equivalence. by 54% over a no-pretraining baseline on a 22-DoF dexterous hand and transferred to robots with simpler hands.

Simulation is the other escape route: GPU-parallel environments compress months of interaction into hours of wall-clock, at the price of the sim-to-real gapsim-to-real gapThe discrepancy between a simulator's physics and rendering and the real world, which makes policies trained in simulation degrade when they are deployed on hardware. The gap has several independent components, dynamics, sensing, and actuation among them, and the standard transfer methods (domain randomization, system identification, teacher-student distillation, real-to-sim reconstruction) each attack a different part of it. covered in massively parallel sim RL. And whichever source the data comes from, the failure modes of learning from it in closed loop are the ones covered in behavior cloning foundations: covariate shiftcovariate shiftIn imitation learning, the mismatch between the state distribution in the training demonstrations and the distribution the learned policy visits at deployment. The policy's own errors carry it into states the demonstrations never covered, where it is likelier to err again, so mistakes compound: Ross, Gordon, and Bagnell showed that a behavior-cloned policy with per-step error ε can incur total cost growing quadratically in episode length, and that no i.i.d.-trained policy can guarantee better. does not care whether the demonstrations came from a teleop rig, a GoPro, or a physics engine.

How to read the field's data claims

Three habits serve well here. First, convert everything to hours: episode counts hide trajectory length, and AgiBot World's million trajectories are 2,976 hours, less than Ego4D's passive video. Second, ask for the diversity breakdown, meaning environments, objects, tasks, and operator styles, because the scaling laws say those axes dominate the hour count. Third, treat vendor-reported scale as a claim until the data or an independent audit is public; AgiBot World's 30% improvement over OXE pretraining AgiBot-World-Contributors 2025↓AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied SystemsAgiBot-World-Contributors, Qingwen Bu, Jisong Cai et al., 2025 is the kind of figure with no independent replication behind it. The bottleneck is real, but so is the incentive to sound past it.