Why RL Won Locomotion but Not Manipulation

Simulation-trained reinforcement learning (RL) made legged locomotion robust well before general manipulation, because walking is cheap to simulate and contact-rich manipulation is not. In 2021 an ANYmal quadruped learned to walk on flat ground in under four minutes, and on uneven terrain in twenty, on one workstation GPU^{Source 1}1 Rudin et al. 2021, Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Nikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021.

Photo: Cvoellmy / Wikimedia Commons. Licence: CC BY-SA 4.0.

The asymmetry is in the MDP

Legged RL left simulation in 2019, when Hwangbo and colleagues transferred trained policies to ANYmal^{Source 8}8 Hwangbo et al. 2019, Learning agile and dynamic motor skills for legged robots. Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy et al., Science Robotics 4(26), 2019. Lee and colleagues' proprioceptive controller then held up in natural environments^{Source 9}9 Lee et al. 2020, Learning Quadrupedal Locomotion over Challenging Terrain. Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen et al., Science Robotics 5(47), 2020, RMA ran on an A1 without fine-tuning^{Source 10}10 Kumar et al. 2021, RMA: Rapid Motor Adaptation for Legged Robots. Ashish Kumar, Zipeng Fu, Deepak Pathak et al., RSS 2021, and an off-policy recipe now trains humanoid locomotion in 15 minutes on one RTX 4090^{Source 11}11 Seo et al. 2025, Learning Sim-to-Real Humanoid Locomotion in 15 Minutes. Younggyo Seo, Carmelo Sferrazza, Juyue Chen et al., 2025.

Boston Dynamics says RL helps where conditions are easy to simulate but good strategies are hard to describe^{Source 12}12 Boston Dynamics 2024, Starting on the Right Foot with Reinforcement Learning. Boston Dynamics, 2024. Rudin and colleagues' policy observes proprioceptionproprioceptionProprioception is sensing the body's own state (joint positions and velocities, orientation, contact), as opposed to exteroception of the outside world through cameras or lidar. It is fast, cheap and never occluded. At ETH Zurich, Lee et al.'s ANYmal controller hiked mud, snow, rubble and vegetation on proprioception alone. Miki et al. kept it as the trusted fallback when the terrain map disagrees with the body. and 108 terrain-height samples, and a learned network models ANYmal's series-elastic actuators, which are hard to simulate^{Source 1}1 Rudin et al. 2021, Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Nikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021. Contact-rich manipulation is harder: the 2026 reality-gap survey notes that most simulators assume rigid bodies while real objects deform, that perfect fidelity is computationally infeasible, and that simplified contact models are particularly problematic in contact-rich tasks^{Source 2}2 Aljalbout et al. 2025, The Reality Gap in Robotics: Challenges, Solutions, and Best Practices. Elie Aljalbout, Jiaxu Xing, Angel Romero et al., Annual Review of Control, Robotics, and Autonomous Systems 2026 (accepted), 2025. Tang and colleagues blame the cost of physical interaction for RL's difficulty in robotics^{Source 13}13 Tang et al. 2025, Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes. Chen Tang, Ben Abbatematteo, Jiaheng Hu et al., Annual Review of Control, Robotics, and Autonomous Systems, 2025.

Locomotion papers combine PPOPPOPPO (proximal policy optimization) is the on-policy reinforcement learning algorithm the locomotion literature standardized on. Schulman and colleagues introduced it as a policy gradient method alternating between sampling environment data and optimizing a surrogate objective by stochastic gradient ascent. The surrogate allows several minibatch epochs per batch where standard policy gradient takes one step. It keeps most of trust region policy optimization's reliability without second-order machinery, and suits massively parallel simulation, whose thousands of robots supply its enormous on-policy batches., domain randomizationdomain randomizationDomain randomization is training a policy across a distribution of simulators with randomized visual appearance, dynamics or both, so the real world reads as one more sample. Tobin et al. randomized rendering and transferred an object detector to a real arm. Peng et al. randomized dynamics and transferred a pushing policy with no real-world training. The cost is conservatism: such a policy is optimal for no single instance. and a curriculumcurriculum learningCurriculum learning is training on tasks that start easy and grow harder as the agent improves. Rudin et al. adjusted each robot's terrain level by its progress. Their documented curriculum-trained policy for simulation and deployment used 4,096 robots, 98,304 transitions per batch and 1,500 updates in under twenty minutes on an i9-11900k CPU and RTX A6000 GPU. Their separate flat-terrain headline is under four minutes on one workstation GPU.. Manipulation papers more often learn from real demonstrations, which need no contact model: Diffusion Policy beat prior methods by 46.9% on average across 12 tasks^{Source 14}14 Chi et al. 2023, Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. Cheng Chi, Zhenjia Xu, Siyuan Feng et al., 2023, and ACT learned to slot a battery at 80 to 90% success from 10 minutes of demonstrations^{Source 15}15 Zhao et al. 2023, Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. Tony Z. Zhao, Vikash Kumar, Sergey Levine et al., RSS 2023.

Six properties of the two MDPs

Boston Dynamics builds Atlas from rotary actuators that are "much easier to represent well in simulation"^{Source 16}16 Rodriguez et al. 2026, Training a Humanoid Robot for Hard Work. Alberto Rodriguez, Shane Rozen-Levy, Vinay Kamidi, Boston Dynamics blog, 2026, and NVIDIA's SimReady asset standard specifies mass, friction, inertia, collision and articulation limits^{Source 17}17 NVIDIA 2026, What Is SimReady?. NVIDIA, NVIDIA Glossary, as of 2026-10-04. Isaac Lab calls multi-fingered manipulation harder than parallel-jaw grasping^{Source 3}3 NVIDIA et al. 2025, Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning. NVIDIA, Mayank Mittal, Yunrong Guo et al., 2025, and Lin and colleagues find sim-to-real dexterous RL still largely limited to single-hand or state-based setups^{Source 4}4 Lin et al. 2025, Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids. Toru Lin, Kartik Sachdev, Linxi Fan et al., CoRL 2025.

Contact geometry is where the gap lives

A gait recovers from a wrong contact force on the next step; a jammed peg cannot. Acosta and colleagues call dynamic contact the computational bottleneck for most simulations, and three simulators they tested missed elastic impacts^{Source 18}18 Acosta et al. 2021, Validating Robotics Simulators on Real-World Impacts. Brian Acosta, William Yang, Michael Posa, arXiv preprint (IEEE RA-L / IROS 2022), 2021. Rigid-body contact models have a task-specific accuracy ceiling^{Source 19}19 Fazeli et al. 2017, Fundamental Limitations in Performance and Interpretability of Common Planar Rigid-Body Contact Models. Nima Fazeli, Samuel Zapolsky, Evan Drumwright et al., 2017, and Factory, which simulates 1000 nut-and-bolt interactions at once, still calls assembly contact a longstanding challenge^{Source 20}20 Narang et al. 2022, Factory: Fast Contact for Robotic Assembly. Yashraj Narang, Kier Storey, Iretiayo Akinola et al., Robotics: Science and Systems (RSS) 2022.

Photo: Richard Greenhill and Hugo Elias, Shadow Robot Company / Wikimedia Commons. Licence: CC BY-SA 3.0.

Exceptions in manipulation

OpenAI's in-hand policy learned finger gaiting without human demonstrations^{Source 21}21 OpenAI et al. 2018, Learning Dexterous In-Hand Manipulation. OpenAI, Marcin Andrychowicz, Bowen Baker et al., 2018. Its Rubik's-cube successor, trained with automatic domain randomization, solved one fixed 43-subgoal scramble in two of ten trials with a sensor cube and zero of ten with vision for face angles, on customized hardware with tuned torque limits and a calibrated hand simulation^{Source 5}5 OpenAI et al. 2019, Solving Rubik's Cube with a Robot Hand. OpenAI, Ilge Akkaya, Marcin Andrychowicz et al., 2019. Visual Dexterity reorients novel shapes in a median of about seven seconds on hardware under $5,000^{Source 22}22 Chen et al. 2022, Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes. Tao Chen, Megha Tippur, Siyang Wu et al., arXiv preprint (Science Robotics 2023), 2022, and DextrAH-RGB transfers an end-to-end RGB grasping policy^{Source 23}23 Singh et al. 2024, DextrAH-RGB: Visuomotor Policies to Grasp Anything with Dexterous Hands. Ritvik Singh, Arthur Allshire, Ankur Handa et al., 2024, distilled from a privileged teacher into a stereo-RGB student for a KUKA arm and Allegro hand^{Source 3}3 NVIDIA et al. 2025, Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning. NVIDIA, Mayank Mittal, Yunrong Guo et al., 2025.

In assembly, AutoMate's generalist solves 20 assemblies zero-shot^{Source 24}24 Tang et al. 2024, AutoMate: Specialist and Generalist Assembly Policies over Diverse Geometries. Bingjie Tang, Iretiayo Akinola, Jie Xu et al., 2024, Play2Perfect reaches 60% on 0.5 mm-clearance insertions^{Source 6}6 Lum et al. 2026, Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?. Tyler Ga Wei Lum, Kushal Kedia, C. Karen Liu et al., CoRL 2026, and InsertAnything, trained only on a simulated hexagonal insertion, succeeded on 95.0% of trials across eight unseen real insertion tasks^{Source 30}30 Ma et al. 2026, InsertAnything: Generalizable Contact-Rich Precision Insertion from Simulation to Reality. Zhenghua Ma, Xinpan Meng, Zeyu Liu et al., 2026. SPARR's real-world residual adds 38.4% success over zero-shot transfer^{Source 25}25 Guo et al. 2026, SPARR: Simulation-based Policies with Asymmetric Real-world Residuals for Assembly. Yijie Guo, Iretiayo Akinola, Lars Johannsmeier et al., 2026.

Such results often need per-task engineering: zero-shot force-controlled dexterous grasping needed tactile simulation, current-to-torque calibration and actuator models^{Source 26}26 Zhao et al. 2026, Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulation. Zhe Zhao, Haoyu Dong, Zhengmao He et al., 2026. SimToolReal instead trains one policy over procedural tools that transfers zero-shot across 24 real tasks^{Source 27}27 Kedia et al. 2026, SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation. Kushal Kedia, Tyler Ga Wei Lum, Jeannette Bohg et al., 2026, and Boston Dynamics says Atlas learned to lift a fridge with perception done implicitly from proprioception^{Source 16}16 Rodriguez et al. 2026, Training a Humanoid Robot for Hard Work. Alberto Rodriguez, Shane Rozen-Levy, Vinay Kamidi, Boston Dynamics blog, 2026.

Imitation-trained baselines

Physical Intelligence reports π0.5 cleaning kitchens and bedrooms in entirely new homes^{Source 28}28 Physical Intelligence et al. 2025, π0.5: a Vision-Language-Action Model with Open-World Generalization. Physical Intelligence, Kevin Black, Noah Brown et al., 2025, and TRI finds that multi-task pretraining raises success and cuts the data a new task needs^{Source 29}29 TRI LBM Team et al. 2025, A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation. TRI LBM Team, Jose Barreiros, Andrew Beaulieu et al., 2025. DeepMind's Gemini Robotics 2 reports 32% to 92% success on multi-finger tasks^{Source 7}7 Parada 2026, Gemini Robotics 2 brings whole body intelligence to robots. Carolina Parada, 2026; humanoid whole-body control has the full results.