RL, Sim-to-Real & Locomotion
Massively Parallel Sim RL
Isaac Lab, Newton, MJX, and Brax: GPU-parallel environments and the wall-clock economics of training.
Last reviewed 2026-08-08
The four-minute training run from the previous module was not an algorithmic result. PPO was already four years old when Rudin and colleagues used it to train ANYmal to walk in minutes Rudin 2021. What changed between 2021 and 2026 was the infrastructure underneath the algorithm: physics moved onto the GPU, environment counts went from dozens to tens of thousands, and the wall-clock cost of a walking policy collapsed by orders of magnitude. This module is about that infrastructure: what the throughput numbers actually are, and where the remaining bottleneck sits.
The 2021 break: physics moved onto the GPU
Isaac Gym was the first demonstration of end-to-end RL for complex robot tasks running entirely on one GPU. PhysX simulated on the GPU and exposed the simulation state directly as PyTorch tensors, so the physics buffers and the policy network never round-tripped through the CPU. The authors report two to three orders of magnitude improvement over the conventional split of a CPU simulator feeding a GPU network Makoviychuk 2021.
Rudin and colleagues turned that capability into the canonical locomotion result. Four thousand parallel ANYmal instances, a game-inspired terrain curriculum (robots that succeed get promoted to harder terrain tiles, robots that fail get demoted), and PPO: flat terrain in under four minutes, uneven terrain in twenty, on a single workstation GPU Rudin 2021. The open-sourced code, legged_gym, is still the reference lineage for the RSL-RL training stack most legged-robot groups use.
The massively parallel regime inverts the usual PPO instincts. Rollouts are short (a few dozen steps per environment per iteration, with bootstrapping covering the rest), batches are enormous (on the order of 100k transitions), and the learning rate adapts to a KL-divergence target rather than following a fixed schedule. None of this works at 16 environments; all of it is stable at 4,096.
The JAX line: Brax, MJX, MuJoCo Playground
NVIDIA's stack was not the only path. Brax, from Google in 2021, wrote the physics and the learning algorithms in JAX so both compile onto the same accelerator, training performant policies on MuJoCo-like tasks in minutes Freeman 2021. MuJoCo XLA (MJX) later brought MuJoCo's contact model into that batched-GPU setting, and MuJoCo Playground (2025) packaged it as an open robot-learning framework spanning locomotion and manipulation, with sim-to-real transfer as a stated goal rather than an afterthought Zakka 2025. Brax remains the JAX-native differentiable option among current engines Cano 2026.
What the environment count actually buys
The economics are worth seeing once. Holding the total experience required to reach a target reward fixed, wall-clock time is iterations times per-iteration cost. Per-iteration cost has a fixed part (the learning update, host-device transfer, the Python loop) and a part that scales with the environment count. At 64 environments the fixed part dominates and the GPU idles; at 16,384 the simulation itself is most of each iteration, and the same training run drops from hours to minutes.
4,096 envs: 4.0 min to target reward at 924k FPS
At low env counts the fixed per-iteration costs (learning update, host-device transfer, the Python loop) dominate and the GPU idles; at high counts simulation takes over and wall-clock falls from hours to minutes. Diamonds are measured wall-clock from Rudin et al. 2021 at 4,096 envs on one workstation GPU.
Two things to try. First, drag the environment slider across its range and watch the breakdown bar recompose: the red CPU-and-transfer share shrinks as parallelism rises. Second, turn on the CPU single-core bottleneck. Isaac Lab's benchmarks carry a systems caveat: on the Franka task, a single-GPU RTX 5090 workstation approaches a 2x RTX PRO 6000 server, because parts of the PhysX pipeline and the main training loop are bound by single-core CPU performance Mittal 2025. The toggle reproduces that regime: once per-environment CPU work exceeds what the GPU can absorb, adding environments stops buying time and the curve flattens.
How to read this chart
The curve is an illustrative fixed-transitions model, not a benchmark; its per-iteration cost split is schematic. The measured numbers are the two diamonds (Rudin et al. 2021: under four minutes on flat terrain, twenty on uneven, at 4,096 environments on one workstation GPU) and the Isaac Lab throughput figures cited below. The direction and shape of the curve, hours at low parallelism to minutes at high, is what every GPU-physics stack reports.
Isaac Lab and the framework era
Isaac Gym was a physics engine with tensors attached. Isaac Lab, its successor, is a full robot-learning framework: OpenUSD as the scene layer, PhysX for physics, RTX for rendering, tiled camera rendering that batches many cameras into one pass, Warp-based raycast LiDAR and height-scan sensors, non-linear actuator models, and a manager-based composable environment API with out-of-the-box hooks into RSL-RL, RL-Games, SKRL, SB3, and Ray Mittal 2025.
The measured throughput is the number that matters here. Benchmarked headless, with distributed training scaling "almost perfectly linearly," Isaac Lab reports over 900,000 frames per second on the state-based DextrAH teacher task and over 1.6 million frames per second on Franka cabinet-opening, both at 8 GPUs and 16,384 environments Mittal 2025. At those rates, a training run that consumed a cluster weekend in 2020 fits in a lunch break.
Isaac Lab 3.0 (2026) then broke the framework loose from its own renderer. The backend-specific code is separated from the core API, so the same environment definition can run on Isaac Sim with PhysX and RTX for photoreal, sensor-rich work, or headless on Newton for raw throughput; photoreal sensors can be reattached through the standalone OVRTX renderer when a vision policy needs them Cano 2026.
Newton: physics as a multi-solver library
Newton, which hit 1.0 GA at GTC in March 2026, is the newest layer. It is an open-source, GPU-accelerated, differentiable physics engine built on NVIDIA Warp and OpenUSD, founded by NVIDIA, Google DeepMind, and Disney Research and governed under the Linux Foundation Reist 2026. The architectural bet is that no single solver wins everywhere, so Newton is a framework of solvers behind one API: MuJoCo Warp (generalized coordinates, the accuracy reference), Featherstone, Kamino from Disney Research (maximal coordinates, for closed-loop linkages like parallel-linkage legs and tendon-driven hands that standard articulation solvers cannot represent), VBD for cables, cloth, and soft bodies (two-way coupled to MuJoCo Warp), and an implicit material-point solver for granular terrain Reist 2026.
The contact story matters more than the solver list. Newton ships SDF-based collision (sparse signed-distance fields precomputed from CAD meshes, with a plus-or-minus 10 mm narrow band) and hydroelastic contacts, a continuous pressure distribution over a finite-area patch, explicitly borrowed from Drake's contact model. These are the pieces aimed at the contact-fidelity problem from the previous module: Skild is using them for GPU-rack connector insertion, and Samsung with Lightwheel for refrigerator hose insertion with the VBD cable solver Reist 2026. Reported throughput: MuJoCo Warp runs 252x faster than MJX on locomotion and 475x on manipulation, measured on an RTX PRO 6000 Blackwell Reist 2026. That figure is vendor-reported, with no independent replication as of mid-2026.
The honest ledger
A few caveats belong with the headline numbers. The Isaac Lab figures come from the framework's own paper, but the hardware, environment counts, and tasks are named, which makes them checkable. The Newton speedups are vendor-reported. NVIDIA itself names the CPU single-core bottleneck as a target for future releases, and domain randomization still writes physics parameters (masses, frictions, contact offsets) through PhysX CPU APIs, which is why randomization happens on episode reset rather than continuously Mittal 2025. Newton's differentiability is advertised, but its practical use for policy learning, as opposed to system identification and design optimization, is early. And Drake keeps its niche: contact-implicit trajectory optimization and rigorous numerics, not throughput Cano 2026.
The next module takes up what happens after the four-minute run finishes: getting the policy out of the simulator and onto hardware, through domain randomization, teacher-student distillation, system identification, and real-to-sim correction.