RL, Sim-to-Real & Locomotion
Why RL Won Locomotion but Not Manipulation
The MDP simulability gap: contact-rich manipulation resists the simulation that made walking routine.
Last reviewed 2026-08-08
In 2021, Rudin and colleagues trained an ANYmal quadruped to walk on flat ground in under four minutes, and on uneven terrain in twenty minutes, on a single workstation GPU Rudin 2021. Policies trained this way are now the default production approach for quadruped locomotion and, increasingly, for bipedal humanoids. General manipulation went the other direction: the field's strongest systems are trained by imitation on real, teleoperated demonstrations. In 2026 there is still no manipulation equivalent of the four-minute walking run.
The asymmetry is in the MDP, not the algorithm
The tempting explanation is that RL is somehow bad at manipulation. The reality-gap survey, the field's authoritative 2026 statement on sim-to-real, points somewhere else: the manipulation MDP is not cheaply simulatable Aljalbout 2025. Locomotion asks the simulator for proprioception, a terrain heightfield, and a decent actuator model, all of which physics engines handle well. Manipulation asks for per-object geometry, friction, compliance, and deformation, resolved to millimeter-accurate multi-point contact, plus a visual pipeline good enough to stand in for object state. Each of those is a known hard problem; a single task needs all of them at once.
This one fact explains why the two literatures read so differently. The locomotion literature is "PPO, domain randomization, curriculum." The manipulation literature is "collect a hundred thousand demonstrations and train a diffusion transformer," because real demonstration data sidesteps the simulability problem entirely: a real trajectory needs no contact model at all (the behavior cloning module covers what that choice costs elsewhere).
Six properties of the two MDPs
The asymmetry decomposes into six specific properties. Each row is a place where the locomotion MDP is cheap to simulate faithfully and the manipulation MDP is not.
| MDP property | Locomotion | Manipulation |
|---|---|---|
| Observation sufficiency | Proprioception (joint positions and velocities, IMU) plus an optional terrain heightscan is nearly sufficient; body state is directly measured. | Object pose, geometry, mass, friction, and deformability are not measured; they must be inferred from pixels. |
| Contact structure | A small number of near-point foot-ground contacts, mostly against a rigid, high-friction substrate. | Many simultaneous, geometrically intricate contacts between non-convex parts at tight tolerance. |
| Sensitivity to contact-model error | High-bandwidth feedback (50-1000 Hz) and a stable gait attractor absorb modeling error. | Insertion and assembly failures are irreversible on a millimeter scale; the policy cannot recover into an attractor. |
| Reward density | Dense: track a commanded body velocity. | Sparse: did the connector seat? |
| Environment authoring cost | Terrain is procedurally generated heightfields, essentially free. | Every task needs new assets with correct physics parameters; "SimReady asset" pipelines exist as a commercial category for this reason. |
| Episode reset | Free in simulation, cheap on hardware. | Real resets need a human or a second robot. |
The Isaac Lab team frames the dexterous case in the same terms: it "remains challenging compared to standard parallel-jaw grasping due to the high-dimensional action space and fine-grained control required," and the framework's answer is SDF collision, accurate contact modeling, and aggressive domain randomization Mittal 2025. The humanoid sim-to-real RL literature is blunter. Lin and colleagues note that previous successes in dexterous manipulation involved "much more laborious real-to-sim engineering effort" than the locomotion successes Lin 2025.
Contact geometry is where the gap lives
Row two of the table deserves its own look, because contact is the property that breaks simulators first. The interactive below injects a contact-model error, the simulator's idea of where a surface is being off by epsilon, into two scenes: a quadruped stance and a peg insertion. At the default error of 2 mm, the quadruped does not notice and the peg is already jammed. Push epsilon past 20 mm and locomotion finally fails, at forty times the error that kills the insertion.
Locomotion: ε = 2.0 mm -> stable: all 4 feet loaded
Quadruped stance: four near-point foot-ground contacts on a rigid substrate. Contact counts and patch radii are illustrative renderings of the asymmetry, not one simulator's solver output. The tolerances are representative physical scales: a gait absorbs centimeter-scale contact error through feedback, while a 0.5 mm insertion clearance makes the same error fatal.
Three things to take from it. First, contact count: four near-point foot contacts versus fourteen simultaneous distributed contacts, each one a place where a solver approximation becomes a wrong force. Second, patch character: a foot presses a near-point patch into a rigid substrate, while a peg slides through continuous wall contact that simple point-contact models do not represent. Third, irreversibility: the quadruped's gait is an attractor that high-bandwidth feedback pulls it back into, so a wrong contact force is corrected on the next step. A jammed peg has no next step.
It is a tendency, not a law
Sim-to-real RL has produced real dexterous results, and honesty requires listing them. OpenAI solved a Rubik's cube with a five-fingered hand in 2019, trained entirely in simulation with automatic domain randomization OpenAI 2019. Isaac Lab's DextrAH line distills a state-privileged RL teacher into a stereo-RGB student, the first system to map stereo images directly to dexterous grasping end to end Mittal 2025. Play2Perfect trains CAD-derived sparse-reward assembly tasks and transfers zero-shot, including 60% success on insertions with 0.5 mm clearance Lum 2026.
What these results share is per-task engineering. The Rubik's cube run consumed a custom robot platform and a randomization curriculum tuned for that task. Play2Perfect's clearance number comes from a pipeline built around its task family. The correct statement of the asymmetry is not that manipulation sim-to-real fails; it is that manipulation sim-to-real requires engineering per task that locomotion gets for free, and that per-task cost is exactly what the MDP table prices.
The honest snapshot comes from the imitation side
The sharpest 2026 evidence that manipulation remains the bottleneck is not an RL result at all. Gemini Robotics 2, the strongest frontier system with published task-level numbers, reports multi-finger household tasks spanning 32% (dustpan) to 92% (unscrewing a bulb), 89.6% on gripper-based precise insertion, and whole-body pick success of 68.4% from a table, 45.7% from the floor, and 76.3% from a shelf Parada 2026. A 32% task on the frontier's best system is what an unsolved MDP looks like.
How to read this module
The interactive's contact counts, patch radii, and scene geometry are illustrative renderings of the asymmetry, not measurements from a specific simulator. The sourced numbers are the Rudin wall-clock figures (under four minutes and twenty minutes), the Gemini Robotics 2 success rates, and the Play2Perfect clearance result. The tolerance values (20 mm absorbable gait error versus 0.5 mm insertion clearance) are representative physical scales, consistent with the cited systems, not a benchmark.
The rest of this domain works through the consequences: the GPU-parallel infrastructure that made the four-minute run possible, the transfer toolbox (domain randomization, teacher-student distillation, system identification, real-to-sim), the locomotion lineage itself, humanoid whole-body control, and the reward-design question that every one of these systems papers over.