RL, Sim-to-Real & Locomotion
Legged Locomotion Lineage
From ANYmal to Unitree and the MIT humanoid line: how learned gaits became the default.
Last reviewed 2026-08-08
The previous modules built the machinery: massively parallel simulation that trains a walking policy in minutes, and the transfer techniques that get it onto hardware. This module is where that machinery was proven. Legged locomotion is the one domain where sim-to-real reinforcement learning is not a promise or a demo but the shipping default, and the chain of results that made it so is unusually well documented. Between 2019 and 2022, the ETH Zurich line around ANYmal took a quadruped from fragile lab behavior to blind alpine hiking, and every humanoid program since has tried to rerun the same recipe on two legs.
The quadruped lineage
The first crack in the problem was not a better algorithm but a better actuator model. ANYmal's series-elastic actuators were the dominant source of sim-to-real error, and Hwangbo and colleagues replaced the analytic model with a learned actuator network: a neural map from joint-command history to realized torque, trained on data from the real robot. With the actuator gap closed, policies trained in simulation produced agile dynamic skills on ANYmal, including recovery from a fall and self-righting Hwangbo 2019.
Lee and colleagues then made locomotion blind and rugged. A privileged teacher policy, trained with simulator-only state, was distilled into a student that sees only a short history of proprioception through a temporal convolutional network. Trained in simple simulated domains, the student hiked mud, snow, rubble, and vegetation it had never seen Lee 2020. The sim-to-real module covers the distillation mechanics; the point here is that the deployed controller perceived no terrain at all and walked anyway.
Miki and colleagues added exteroception back, carefully. A height scan of the terrain ahead helps only until it lies: snow, vegetation, and water read as untraversable obstacles or vanish from the map entirely. Their controller feeds the scan through an attention-based recurrent encoder that learns a belief over the terrain, integrating proprioception so the policy can discount the map when the two disagree. The result completed an hour-long hike in the Alps in the time recommended for human hikers Miki 2022.
Rudin and colleagues collapsed the cost of the whole pipeline. Massive GPU parallelism plus a game-inspired terrain curriculum, where robots that succeed are promoted to harder terrain and robots that fail are demoted, trained ANYmal to walk on flat ground in under four minutes and on uneven terrain in twenty, on a single workstation GPU Rudin 2021. The open-sourced legged_gym code became the reference implementation that most subsequent legged-RL work, academic and industrial, builds on.
Choi and colleagues extended the recipe to ground that moves. Deformable terrain like sand defeats rigid-contact simulators, so they put a computationally cheap granular-media model inside the training loop and paired it with a controller that identifies terrain properties from feel as the robot walks. Their Raibo quadruped ran on soft beach sand at 3.03 m/s with its feet fully buried during stance Choi 2023.
A gait is a timing pattern
Underneath all of this is an older idea that learning did not replace: the gait. A quadruped's gait is a footfall-timing pattern, described by two numbers per leg: the phase offset (when in the stride cycle the foot strikes) and the duty factor , the fraction of the cycle the foot spends on the ground. The diagram below shows the four canonical patterns.
The walk spreads footfalls evenly and keeps three feet down at all times, so a static support polygon always exists. The trot, the working gait of most quadruped robots including ANYmal, strikes diagonal pairs together and balances dynamically on two feet. The bound alternates the front pair against the hind pair with a suspension interval in between; the MIT Cheetah line made high-speed bounding practical by scaling the duty cycle with speed Park 2017. The pronk throws all four feet down and up together and spends most of the cycle airborne.
walk at 0% feet down RF + LH + RH duty 0.75
Four-beat gait; three feet stay down at all times. Amber blocks mark the feet touching the ground at the playhead. Step through the cycle or press play; the walk never drops below three feet of support, the trot balances on diagonal pairs, and the bound and pronk spend part of every cycle airborne.
Switch from walk to trot to bound and watch the support readout: three feet always down, then two, then a flight phase with none. That progression is the stability story of legged locomotion in miniature. Learned policies are not handed these patterns. Foot-air-time and footfall-style reward terms induce them, and the policy discovers whichever pattern the reward makes cheapest. The duty factors shown here are canonical nominal values; real controllers, classical and learned alike, modulate duty factor continuously with speed Park 2017.
From quadrupeds to humanoids
The recipe moved to two legs on two hardware lines. Unitree's H1 and G1 became the default research platforms: H2O turned retargeted human keypoints into real-time whole-body teleoperation on the H1, with RL supplying the physical feasibility that raw retargeting destroys He 2024. ASAP added a delta-action correction stage, trained on real rollouts and injected back into the simulator, and produced jumps and sports-style motions on the G1 He 2025.
MIT's humanoid program attacked a different piece of the problem. On the MIT humanoid platform, Jeon and colleagues benchmarked potential-based reward formulations for learned locomotion, showing systematically how the shaping of the reward, not the choice of RL algorithm, decides whether training converges to a usable gait Jeon 2023. That finding generalizes uncomfortably well: across the whole lineage, the algorithm is the commodity and the objective design is the work. The full human-motion-tracking lineage, from PHC through GMT and the 2026 robustness results, gets its own module next.
Industrial adoption
Boston Dynamics' own history is the cleanest industrial verdict. Spot's production locomotion controller was MPC-based: dozens of predictive controllers evaluated in parallel and scored every step. In 2024 the company integrated a learned policy into that stack to handle the real-world variability that hand-scored MPC struggled with, explicitly as an addition rather than a replacement. The hybrid system reduced onboard compute (no more parallel MPC instances), cut fall rates on slick and irregular surfaces, and, freed from periodic-gait assumptions, climbs boxes over 70 cm. The engineering wrapper is telling: policies are vetted on a 24/7 robustness fleet logging over 2,000 hours a week before they ship, and reproducible falls are recreated in simulation and folded back into training Dynamics 2024.
Atlas moved in two steps. In early 2025 Boston Dynamics and the RAI Institute showed RL policies on the new electric Atlas that track retargeted human motion, each maneuver distilled from roughly 150 million simulator runs and transferred zero-shot to hardware Institute 2025. By August 2025 the collaboration with Toyota Research Institute had shifted Atlas's manipulation behaviors to Large Behavior Models: a 450M-parameter diffusion transformer with a flow-matching objective, language-conditioned, controlling the full robot at 30 Hz from teleoperated demonstrations Cousineau 2025. Neither step threw out the model-based stack; Spot's MPC still runs underneath, and Atlas's teleoperation system is built on its MPC. The accurate framing for 2026 is hybrid, not either/or.
What the lineage does not solve
Hold onto three limits. First, these controllers walk, and walking is not fetching: nothing in the legged success story transfers directly to contact-rich manipulation, where the first module in this domain showed the MDP itself resists simulation. Second, motion tracking inherits the human demonstrator's envelope, and retargeted references can be morphologically infeasible for the robot, which is the explicit motivation of the 2026 robustness work the humanoid module covers. Third, the lineage's center of gravity turned out to be reward design: the gait a policy learns is whichever pattern the reward makes cheapest, and getting that objective right is still hand labor, now with LLM assistants. The final module in this domain takes up reward shaping and the unresolved debate with classical MPC.