RL, Sim-to-Real & Locomotion
Sim-to-Real Transfer
Domain randomization, teacher-student distillation, system identification, and real-to-sim correction.
Last reviewed 2026-08-08
The previous module ended with a policy that walks after minutes of GPU training. That policy is useless as trained: it has only ever seen one simulator, with one set of contact parameters, one actuator model, one rendering stack. The real robot differs from all of them in ways nobody can fully enumerate. Closing that difference is sim-to-real transfer, and it is where most of the engineering lives. The RL algorithm is usually the least interesting part of a successful sim-to-real system.
Four method families do the work. Domain randomization trains over a distribution of simulators so the real world reads as one more sample. Teacher-student distillation trains on privileged simulator state, then compresses that knowledge into a policy that sees only what the robot sees. System identification measures the dynamics gap and corrects the model directly. Real-to-sim reconstruction rebuilds the deployment scene inside the simulator so the visual gap shrinks to rendering error. The 2026 reality-gap survey organizes the field into nearly these same buckets, adding sim-real co-training and state/action abstraction as cross-cutting levers Aljalbout 2025.
Domain randomization: train on a distribution of simulators
The idea dates to two 2017 results. Tobin and colleagues randomized rendering (textures, lighting, camera pose) and showed a detector trained only on non-realistic simulated images could localize objects to 1.5 cm on a real arm Tobin 2017. Peng and colleagues randomized dynamics instead (mass, inertia, friction) and transferred a pushing policy to hardware with no real-world training, robust to significant calibration error Peng 2018. With enough variability in the simulator, the real world may appear to the model as just another variation, as Tobin's abstract puts it.
OpenAI's Rubik's-cube system added the curriculum version: automatic domain randomization widens the randomization range as the agent succeeds, so the distribution grows with the policy's competence OpenAI 2019. Isaac Lab ships ADR as a configurable curriculum with reference configs in its dexterous-manipulation examples Mittal 2025.
Two details rarely make it into the textbook version. First, the plumbing: in Isaac Lab, simulation state is GPU-resident but simulation parameters (masses, frictions, contact offsets, joint armature) must still be written through PhysX CPU APIs, and mesh scale or collider type can only change before the simulator starts Mittal 2025. This is why randomization happens on episode reset rather than continuously, and it is a systems constraint, not a theoretical one. Second, the cost: randomizing over a wide dynamics distribution produces a policy that is optimal for none of them. DR buys a wider basin of transfer at the price of peak performance, and papers almost never quantify the price.
real mu 0.80: point 97% vs DR 74%
The real robot sits inside the training distribution, where the point-trained policy wins: specializing at one friction bought it a higher peak than any robust policy reaches. Move the line outside the shaded band and the ranking flips.
The interactive makes the trade concrete. The gray spike is a policy trained at one friction; the amber plateau is a policy trained over the shaded distribution. Drag the real-robot line off the training point and the ranking inverts: the specialist collapses, the generalist holds. Then widen the randomization range and watch the plateau sink. Coverage and peak pull against each other, and the knob only moves along that frontier.
How to read this chart
Both curves are an illustrative model, not a benchmark. The sourced facts are the framing (Tobin and Peng's randomization results, ADR's widening curriculum, the CPU-API constraint) and the qualitative shape, which every DR paper reports: specialists win at the training point and lose away from it. The specific success values are schematic.
Teacher-student distillation: privilege in, proprioception out
Domain randomization still asks the policy to act on what the robot can sense. The teacher-student pattern separates those problems. Phase one trains a teacher with privileged, simulator-only observations: terrain geometry under the feet, friction coefficients, applied disturbances. Phase two trains a student that sees only deployable sensing, usually a short history of proprioception, to imitate the teacher's actions.
The reference result is Lee and colleagues' ANYmal controller: a privileged teacher distilled into a temporal convolutional network over proprioceptive history, trained in simple simulated domains and deployed blind on mud, snow, rubble, and vegetation it never saw in training Lee 2020. RMA factorized the same idea into a base policy conditioned on a latent extrinsics vector plus an adaptation module that regresses those extrinsics online from recent proprioception, adapting to new payloads and terrains in fractions of a second at control rate Kumar 2021. The adaptation module is, in effect, system identification performed in latent space by the policy itself.
The pattern has since moved to manipulation. Isaac Lab's DextrAH-RGB distills a state-privileged RL teacher into a stereo-RGB student, which the authors describe as the first system to map stereo images directly to dexterous grasping end to end Mittal 2025.
The failure mode has a name. Isaac Lab calls it an information-capacity mismatch: the teacher observes privileged state while the student sees partial observations and must reconstruct the unobserved rest, a gap that becomes pronounced under camera occlusion Mittal 2025.
degradation 15%: MAE 0.01 m divergence 0.02
Brighter cells are higher terrain; the teacher sees them directly. The student sees only the bar strip: joint positions, velocities, and contact events over a short history. At zero degradation its reconstruction matches the teacher's map and the two policies agree. As noise and occlusion rise, the reconstruction blurs, the student's belief about what is under the feet drifts, and its actions diverge from the privileged teacher's.
Push the degradation slider up. The student's input strip loses channels and gains noise, its reconstructed terrain blurs away from the teacher's map, and the action-divergence readout climbs. That divergence is the cost of distillation, and its floor is set by partial observability rather than by training budget: when the privileged variable genuinely cannot be inferred from the history (heavy occlusion, friction that leaves no proprioceptive signature), no amount of imitation closes the gap.
System identification: measure the gap instead
Randomization averages over model error. System identification removes the error by measurement. The ancestral version in this lineage is Hwangbo and colleagues' actuator network: a learned map from joint-command history to realized torque, replacing the analytic actuator model whose miscalibration had been the dominant sim-to-real error source on ANYmal Hwangbo 2019.
The cleanest modern formulation is ASAP. Pre-train a motion-tracking policy in simulation on retargeted human motion, roll it out on the real Unitree G1, then train a delta action model: a network that maps simulator state and commanded action to the residual action that would have reproduced the observed real transition. Insert the delta model into the simulator and fine-tune the policy against the corrected dynamics He 2025. The loop runs sim to real to sim to real, and ASAP evaluates it across IsaacGym-to-IsaacSim, IsaacGym-to-Genesis, and IsaacGym-to-hardware transfers, reducing tracking error against SysID, DR, and delta-dynamics baselines He 2025.
The caveat is baked into the construction. The delta model is learned on the state distribution the pre-trained policy visits, so it is only valid near that distribution; the fine-tuned policy cannot stray far before the correction itself becomes unreliable He 2025.
Real-to-sim: rebuild the scene, keep the physics
The newest family attacks the visual gap by reconstructing the actual deployment scene. SplatSim replaces the simulator's mesh renderer with a 3D Gaussian Splatting reconstruction of the real environment and reports zero-shot transfer of RGB manipulation policies at 86.25% average success, against 97.5% for policies trained on real data Qureshi 2024. RoboGSim packages the same loop as a real2sim2real simulator: reconstruct the scene as splats, compose novel views, objects, and trajectories, and evaluate policies online against the twin Li 2024. Newton 1.0's tiled camera sensor accepts Gaussian splats as a native scene representation, which is this idea productized Reist 2026.
Keep the division of labor straight: the splat supplies appearance, and a conventional physics engine still supplies dynamics. A 3DGS twin is a learned renderer bolted onto a simulator, not a learned simulator, and it reconstructs a static scene: articulated objects, deformables, and lighting changes each require extra machinery on top. Confusing the two is the most common misreading of the real-to-sim literature.
The levers nobody demos
The survey adds two cross-cutting options that rarely get headline systems. Sim-real co-training mixes real data into an otherwise simulated training set, so the real samples anchor the representation while simulation supplies coverage. State and action abstraction chooses the interface between policy and robot (end-effector velocities or foot positions rather than raw torques) so that a conventional low-level controller absorbs much of the gap Aljalbout 2025. The survey calls the abstraction choice one of the highest-leverage and least-discussed levers in the field, which matches the quiet pattern in successful deployments: most of them randomize some things, identify others, and abstract away the rest.
Where each family breaks
Each family's failure mode is worth holding onto. Domain randomization's conservatism cost is real and almost never quantified in the papers that rely on it. Teacher-student distillation bottoms out at partial observability when the privileged variable leaves no trace in the student's history. Delta-action correction is valid only near the trajectories it was fit on. Real-to-sim twins freeze the scene they captured. The next module, on the legged locomotion lineage, shows these techniques stacked in the field's one complete success story, from Hwangbo's actuator networks through ANYmal's blind hiking controller to the minutes-scale training pipelines built on top of them.