A robot-learning system is a pipeline from physical events to decisions and back again; the checkpoint is one component among capture, schemas, training, evaluation, serving and recovery. The open-source LeRobot library spans that stack, from motor middleware to dataset storage, streaming and asynchronous inferenceSource 10.

The OpenVLA overview figure: a montage labelled 970k Robot Episodes feeds a vision-language-action model built on Llama 2 7B, above photos of three robots it controls, a WidowX at a toy sink, a mobile robot by an apple and a banana, and a Franka with a dustpan.
OpenVLA's overview traces one path through the stack: 970k robot episodes, a 7B vision-language-action model and closed-loop control of several robots.
Paper figure: Moo Jin Kim et al. / arXiv. Licence: CC BY 4.0.

Machine learning systems often incur massive ongoing maintenance costs from data dependencies, configuration issues and hidden feedback loopsSource 12. Splitting the stack into seven contracts, each with an artifact and a test, lets a failure be traced to one boundary. Try the check below; video alone only replays an episode.

Self-check

Which dataset record is sufficient to reconstruct one policy transition after the camera mount and training code change?
Read the reasoning
  • A compressed video and a final success labelThis preserves appearance but loses robot state, action timing, transforms and the command that produced each transition.
  • The normalized, synchronized tensors consumed by one training runTensors help exact reruns but hide how units, frames and synchronization were derived. They cannot be reprocessed safely when the schema changes.
  • Versioned raw streams with actions, transforms and calibrationThis preserves the evidence needed to reconstruct features, audit alignment and migrate preprocessing while keeping provenance.

Store raw, timed and versioned context to rebuild a transition; pixels alone let you watch it.

Contract 1: hardware and calibration

Record firmware, control mode and calibrated transforms for every camera, encoder, gripper and sensor: a dataset split cannot correct an unrecorded camera move. Time offsets between sensors are calibration parameters too, estimable together with the spatial onesSource 13, and calibration covers the residuals.

A grid of DROID camera frames from different scenes, each with a purple mask of the robot arm rendered over the real arm in the image.
DROID frames with the robot's mesh rendered from the estimated camera-to-base transform. Where the mask covers the real arm, the calibration holds; a moved camera shows as a mask that misses.
Paper figure: Alexander Khazatsky et al. / arXiv. Licence: CC BY 4.0.

Contract 2: episode capture

An episode holds raw observations, robot state, commanded action, measured response, outcome, interventions and timestamps. ROS 2 Iron made MCAP its default bag format after testing found SQLite3 storage often not performant enoughSource 14. RLDS is a standard, lossless format for sharing episodesSource 15. Keep resets and failures: RoboMIND releases 5,000 real-world failure demonstrations with annotated causesSource 16.

DROID standardized collection hardware and protocols across institutions to gather 76k in-the-wild trajectoriesSource 2; Open X-Embodiment chose breadth, normalizing over one million trajectories across 22 embodimentsSource 1.

Contract 3: a versioned training schema

Every preprocessing step, from frame selection to normalization and splits, needs a version. LeRobot provides datasets, policy implementations and tools around a common workflowSource 3. Its v3.0 format packs many episodes into each file after one file per episode hit file-system limits at millions of episodesSource 17.

Joint targets, Cartesian deltas and torques are different action labels even at equal array widths. Per-dimension binning tokenizers perform poorly on high-frequency dexterous data, and FAST+ is a universal tokenizer trained on 1M real trajectoriesSource 18.

Split by the factor a claim generalizes across: random episode splits leak near-identical scenes. Across 15,000+ real rollouts, environment and object diversity mattered far more than demonstration countSource 19.

Splitting by scene
Test on scenes the model never trained on
The split
Adjust more
Raised against the lit rail: the test set; a card's shape is its sceneRandom episodes · test cards from 3 scenes · all 3 also in training

Schematic: 20 illustrative episodes and one random draw

Split by the factor you claim to generalize across: a random split lets every test scene into training.
How this was made

The article's rule for the training schema: split by the factor you claim to generalize across, because random episode splits can leak near-identical scenes and operators. Hold out environments for environment generalization, tasks for task generalization and embodiments for cross-embodiment claims.

The rack is schematic: 20 episodes, 5 recorded in each of 4 scenes, and a test set of 5. The random split is one seeded draw: it takes test episodes from 3 scenes, and each of them keeps episodes in training. That is not luck. Of the 15,504 ways to draw 5 test episodes from 20, only 4 take a whole scene, so nearly every random split tests on scenes the model also trained on. Holding out scene 4 is one of those 4, chosen on purpose.

This split: test scenes 1, 2, 4; also in training: 1, 2, 4.

Contract 4: reproducible training

A training run saves configuration, code revision, dataset hash, seeds and normalization statistics with its weights. In robomimic's study of offline human demonstrations, recurrent behavior cloning was strong and the batch RL algorithms struggled to learn from the same dataSource 4.

OpenVLA is a public seven-billion-parameter VLA adapted on standardized robot trajectoriesSource 8, and the OpenVLA-OFT recipe raised its average LIBERO success from 76.5% to 97.1% and its action throughput 26-foldSource 20. Physical Intelligence reports RECAP, which learns from deployment experience, more than doubles throughput on some of its hardest tasksSource 21.

Contract 5: simulation and offline evaluation

Isaac Lab supports large batches of robot-learning environments and sensor modalitiesSource 5. Simulated scores help only when they rank policies as hardware does, which SIMPLER measures on matched tasksSource 6. PolaRiS rebuilds real scenes from short video scans and reports much stronger correlation with real performance than existing simulated benchmarksSource 22. SimFoundry reports a mean Pearson correlation of 0.911 across 7 tasksSource 23.

Co-training on simulated and real data raised real-world task performance by 38% on averageSource 24.

Contract 6: real-world evaluation

Papers often report success rates with little or no information on the number of runs, initial conditions or success criteriaSource 25. LIBERO-Plus showed controlled perturbations across seven dimensions reveal robustness gaps that impressive benchmark success rates can maskSource 7; the evaluation crisis covers the statistics.

RoboArena ranks generalist policies through double-blind pairwise episodes run at several universitiesSource 26. AutoEval runs evaluations around the clockSource 27, and RoboDojo pairs 42 simulated and 18 real-world tasksSource 28. A sequential test that can stop early cut evaluation burden by up to 70% against batch methodsSource 29.

Contract 7: serving and robot integration

Production inference needs an observation assembler, model server, command gate, controller bridge and supervisor, with safety and high-rate feedback below the learned policy. ROS 2 was redesigned from the ground up because ROS 1 lacked production-grade featuresSource 30, and the ROS 2 guide defines the interfaces. For the controller bridge, ros2_control is a ROS 2 framework for robot control, including real-time control. Its packages rewrite the ros_control packages of the original ROS to make new hardware simpler to integrateSource 39. In September 2026 Intrinsic released Intrinsic Core, ROS-compatible real-time control, motion planning and Gazebo simulation under Apache 2.0Source 31. The same month, Qualcomm agreed to acquire PickNik, the longtime steward of MoveItSource 32.

Real-time chunking generates the next action chunk during execution, avoiding pauses at chunk boundariesSource 33, and SmolVLA decouples action prediction from executionSource 34; real-time execution has the latency budgets. Exploy, which the RAI Institute open-sourced on October 7, 2026, exports a PyTorch policy's observation code, network and action processing into one ONNX file for a C++ runtime. It checks numerical equivalence before hardware runs, and the institute says it already deploys policies on Spot, Atlas and the Unitree G1Source 9.

The data loop after deployment

In imitation learning, future observations depend on the policy's own actions, breaking the i.i.d. assumption of statistical learningSource 35. Failures should return to the dataset tagged with their cause. Sirius re-weights samples by approximated human trust and reports 27% higher real-hardware success than prior methodsSource 36. CUPID shows under 33% of curated data can yield state-of-the-art diffusion policies on RoboMimicSource 37.

Mulligan, from Shuran Song, Chelsea Finn and colleagues at Stanford (October 2026), starts each round of supervised episodes at observed failures and untried object placements. With value-based action selection it raised final success on three real tasks by 10 to 34 percentage points, scored on 2,550 held-out, blinded episodesSource 38.

A minimal stack for a small team

Start with versioned episode files that are never edited in place, one versioned conversion command, a checkpoint registry, a deterministic offline test batchSource 17 Source 12, an evaluation sheet linked to episode IDsSource 25, and a policy process kept apart from command gatingSource 10 Source 11.