Massively Parallel Sim RL

The four-minute training run from the previous module was not an algorithmic result. PPOPPOProximal policy optimization, the on-policy reinforcement learning algorithm the locomotion literature standardized on. Schulman and colleagues introduced it as a policy gradient method that alternates between sampling data through interaction with the environment and optimizing a surrogate objective with stochastic gradient ascent, where the surrogate is what allows several epochs of minibatch updates on each batch of samples instead of the single gradient step standard policy gradient takes. It keeps most of trust region policy optimization's reliability without the second-order machinery, and it pairs naturally with massively parallel simulation, where thousands of robots supply the enormous on-policy batches it consumes. was already four years old when Rudin and colleagues used it to train ANYmal to walk in minutes Rudin 2021↓Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement LearningNikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021. What changed between 2021 and 2026 was the infrastructure underneath the algorithm: physics moved onto the GPU, environment counts went from dozens to tens of thousands, and the wall-clock cost of a walking policy collapsed by orders of magnitude. This module is about that infrastructure: what the throughput numbers actually are, and where the remaining bottleneck sits.

The chart below illustrates the cost arithmetic before the history. Drag the parallel environments slider from 64 upward: in the default model, fixed CPU-and-transfer work takes a smaller share of each iteration and the chosen experience budget takes less wall-clock time. This is a constructed comparison, not repeated measurements of one training run.

The 2021 break: physics moved onto the GPU

Isaac Gym ran an end-to-end GPU accelerated training pipeline for complex robot tasks on a single GPU. PhysX simulated on the GPU and exposed the simulation state directly as PyTorch tensors, so the physics buffers and the policy network never round-tripped through the CPU. The authors report two to three orders of magnitude improvement over the conventional split of a CPU simulator feeding a GPU network Makoviychuk 2021↓Isaac Gym: High Performance GPU-Based Physics Simulation for Robot LearningViktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo et al., 2021.

Rudin and colleagues report flat-terrain walking policies trained in under four minutes on a single workstation GPU. Separately, their documented simulation-and-deployment policy used 4,096 parallel ANYmal robots, batches of 98,304 RL transitions (24 steps per robot), and 1,500 policy updates in under twenty minutes on an i9-11900k CPU and NVIDIA RTX A6000 GPU. Their game-inspired terrain curriculumcurriculum learningTraining on a scheduled sequence of tasks that grow harder as the agent improves, instead of sampling the full difficulty range from the start. Rudin and colleagues raised or lowered terrain levels based on each robot's progress. Their documented curriculum-trained simulation-and-deployment policy used 4,096 robots, 98,304 transitions per batch and 1,500 updates in under twenty minutes on an i9-11900k CPU and RTX A6000 GPU; their separate flat-terrain headline is under four minutes on one workstation GPU. promotes robots that cross a tile boundary, demotes those that cover less than half the commanded distance, and returns robots that solve the hardest level to a random level Rudin 2021↓Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement LearningNikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021. The paper links its released training code, legged_gym; the pinned October 2021 repository identifies itself as the Isaac Gym environment used to train ANYmal on rough terrain Rudin 2021↓Isaac Gym Environments for Legged RobotsNikita Rudin, 2021.

The massively parallelparallel simulationRunning thousands of physics simulator instances at once on a single GPU so a reinforcement learning agent collects experience at a rate no CPU cluster matches. Isaac Gym made the setup practical by keeping both physics and policy training on the GPU and passing data straight from physics buffers to PyTorch tensors, skipping the CPU round-trip entirely, which bought two to three orders of magnitude in throughput over a CPU simulator feeding a GPU learner. Rudin and colleagues showed what that buys: four thousand parallel ANYmal instances trained with PPO learned flat-terrain walking in under four minutes and uneven terrain in twenty, on one workstation GPU. regime inverts the usual PPO instincts. Rollouts are short (a few dozen steps per environment per iteration, with bootstrapping covering the rest), batches are enormous (on the order of 100k transitions), and the learning rate adapts to a KL-divergence target rather than following a fixed schedule. None of this works at 16 environments; all of it is stable at 4,096.

The JAX line: Brax, MJX, MuJoCo Playground

NVIDIA's stack was not the only path. Brax, from Google in 2021, wrote the physics and the learning algorithms in JAX so both compile onto the same accelerator, training performant policies on MuJoCo-like tasks in minutes Freeman 2021↓Brax: A Differentiable Physics Engine for Large Scale Rigid Body SimulationC. Daniel Freeman, Erik Frey, Anton Raichuk et al., 2021. MuJoCo XLA (MJX) later brought MuJoCo's contact model into that batched-GPU setting, and MuJoCo Playground (2025) packaged it as an open robot-learning framework spanning locomotion and manipulation, with sim-to-real transfer as a stated goal rather than an afterthought Zakka 2025↓MuJoCo PlaygroundKevin Zakka, Baruch Tabanpour, Qiayuan Liao et al., 2025.

What the environment count actually buys

This authored fixed-transitions model is not a benchmark. We choose a budget of 220 million transitions and 24 rollout steps per environment per iteration, with 4,096 environments and the CPU toggle off by default. Each iteration costs 0.02 seconds plus 0.000004 seconds per environment for simulation, 0.04 seconds for learning, and 0.03 seconds for CPU and transfer work. The toggle adds 0.000022 seconds per environment to the CPU bucket. These costs and the fixed budget are illustrative choices, not estimates recovered from either paper. The curve uses 49 logarithmically spaced environment counts from 64 to 16,384, rounded to integers, and sample times rounded to 0.01 minute. Its near-four-minute default is a chosen calibration, not validation by the flat-terrain result. The flat diamond also uses a chosen 4,096-environment position; only the separate rough-run protocol establishes that count.

The CPU toggle illustrates a possible bottleneck; it does not reproduce a measured Isaac Lab curve. In the paper’s state-based Franka cabinet-drawer benchmark, a GeForce RTX 5090 workstation with an AMD 9800X3D CPU approaches a two-GPU RTX PRO 6000 server with two AMD EPYC 9554 CPUs. The authors attribute the task-dependent comparison to CPU bottlenecks in parts of PhysX and the main training loop, together with the workstation’s fast single-core CPU. This is a finding about those systems, not a universal CPU limit
Source: NVIDIA 2025↓Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot LearningNVIDIA, Mayank Mittal, Yunrong Guo et al., 2025.

Isaac Lab and the framework era

The 2025 Isaac Lab v1 paper describes OpenUSD scene graphs, PhysX simulation, RTX rendering, and camera tiling that batches cameras into one render pass. Warp ray-casting supplies height-scanner and LiDAR-like distance queries; those geometric sensors omit photorealistic rendering effects. The framework includes implicit and explicit actuator models, including nonlinear effort limits and learned actuator networks, plus a manager-based environment API and built-in support for RSL-RL, RL-Games, SKRL, Stable-Baselines3, and Ray
Source: NVIDIA 2025↓Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot LearningNVIDIA, Mayank Mittal, Yunrong Guo et al., 2025.

In the paper’s headless state-based benchmarks, distributed training on RTX PRO 6000 GPUs scales "almost perfectly linearly" over the tested configurations. At the reported eight-GPU, 16,384-environment setting, the DextrAH teacher grasp-and-lift task exceeds 900,000 frames per second and the Franka cabinet-drawer task exceeds 1.6 million. These are environment-learning FPS: environment steps divided by simulation plus learning time, not camera-only FPS or a robot control frequency. The RTX PRO 6000 server uses two AMD EPYC 9554 64-core processors
Source: NVIDIA 2025↓Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot LearningNVIDIA, Mayank Mittal, Yunrong Guo et al., 2025.

NVIDIA's 2026 overview describes Isaac Lab 3.0 as separating backend-specific code from the core API. Developers can use Isaac Sim with PhysX and RTX for photorealistic, sensor-rich workflows, or run headless Newton physics for high-throughput simulation. The overview also describes adding photoreal sensors through the standalone OVRTX renderer, or using Newton's renderer for vision-based RL with many environments Cano 2026↓The State of Simulation for Physical AI: An OverviewJohnny Nuñez Cano, Mitesh Patel, Asier Arranz et al., 2026.

Newton: physics as a multi-solver library

NVIDIA's March 16, 2026 release post announces Newton 1.0 GA at GTC 2026 and describes an open-source, GPU-accelerated physics engine built on NVIDIA Warp and OpenUSD. It identifies Newton as a Linux Foundation project founded by NVIDIA, Google DeepMind, and Disney Research. The post calls Isaac Sim 6.0 and Isaac Lab 3.0 early access releases Reist 2026↓Newton Adds Contact-Rich Manipulation and Locomotion Capabilities for Industrial RoboticsPhilipp Reist, Miguel Zamora Mora, JC Chang et al., 2026. The later NVIDIA overview describes the engine as differentiable, while noting that differentiation support differs between solvers Cano 2026↓The State of Simulation for Physical AI: An OverviewJohnny Nuñez Cano, Mitesh Patel, Asier Arranz et al., 2026.

Newton puts multiple solvers behind a unified API. NVIDIA's overview lists MuJoCo and Featherstone for generalized-coordinate articulated rigid bodies; SemiImplicit, XPBD, and Kamino for maximal-coordinate formulations; VBD for rigid bodies, particles, cloth, and soft bodies, with limited joint support; implicit MPM for particle-based continuum materials; and Style3D for cloth Cano 2026↓The State of Simulation for Physical AI: An OverviewJohnny Nuñez Cano, Mitesh Patel, Asier Arranz et al., 2026. The release post describes Disney Research's Kamino as handling closed-chain mechanisms, and describes coupling VBD and iMPM with MuJoCo Warp for deformable manipulation and locomotion Reist 2026↓Newton Adds Contact-Rich Manipulation and Locomotion Capabilities for Industrial RoboticsPhilipp Reist, Miguel Zamora Mora, JC Chang et al., 2026.

The release post describes SDF-based collision built from CAD-exported meshes and hydroelastic contacts inspired by Drake: pressure is distributed across finite-area contact patches rather than isolated points. Its assembly code example precomputes sparse SDFs with narrow_band_range=(-0.01, 0.01), a plus-or-minus 10 mm band around the surface. That band belongs to the example configuration, not a stated engine-wide requirement Reist 2026↓Newton Adds Contact-Rich Manipulation and Locomotion Capabilities for Industrial RoboticsPhilipp Reist, Miguel Zamora Mora, JC Chang et al., 2026.

For GPU rack assembly, NVIDIA says Skild AI is training RL policies with Isaac Lab's Newton backend, using SDF collision and hydroelastic contact modeling for tasks including connector insertion. The same post says Samsung "will use Newton" for synthetic data generation to train VLA models and describes Lightwheel's work on calibrated SimReady assets. Its refrigerator-assembly example is a simulated RB-Y1 cable-insertion task using two-way coupled MuJoCo Warp and VBD; the text describes inserting a water-hose connector into its housing Reist 2026↓Newton Adds Contact-Rich Manipulation and Locomotion Capabilities for Industrial RoboticsPhilipp Reist, Miguel Zamora Mora, JC Chang et al., 2026. These are NVIDIA's accounts of the workflows, not independently established production-deployment results.

In its MuJoCo 3.5 (MJWarp) release summary, NVIDIA reports speedups over MJX of 252x for locomotion and 475x for manipulation on NVIDIA RTX PRO 6000 Blackwell Series hardware Reist 2026↓Newton Adds Contact-Rich Manipulation and Locomotion Capabilities for Industrial RoboticsPhilipp Reist, Miguel Zamora Mora, JC Chang et al., 2026. The post does not specify benchmark task variants, environment counts, numeric precision, or the timing definition for those ratios. Do not read them as end-to-end policy-training-time or control-frequency measurements.

The honest ledger

A few caveats belong with the headline numbers. The Isaac Lab figures come from the framework's own paper, but the hardware, environment counts, and tasks are named, which makes them checkable. The Newton speedups are vendor-reported.

The v1 paper identifies CPU bottlenecks in its tested systems as a target for future releases. For domain randomizationdomain randomizationTraining a policy across a distribution of simulators with randomized parameters, visual appearance, dynamics, or both, so the real world reads as one more sample from that distribution. Tobin et al. randomized rendering and transferred an object detector to a real arm; Peng et al. randomized dynamics and transferred a pushing policy with no real-world training. The price is conservatism: a policy that does well across a wide distribution is optimal for no single instance of it., it distinguishes GPU-resident simulation state from physics parameters such as mass, friction, contact offsets, and joint armature, which use PhysX CPU APIs. Most of those parameters can change at runtime; mesh scale and collider type can only be randomized before simulation starts
Source: NVIDIA 2025↓Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot LearningNVIDIA, Mayank Mittal, Yunrong Guo et al., 2025.

Newton's differentiability is advertised, but its practical use for policy learning, as opposed to system identificationsystem identificationMeasuring a real robot's dynamics and correcting the simulator's parameters to match, so the sim-to-real gap shrinks by calibration rather than by randomization. Hwangbo and colleagues replaced a miscalibrated analytic actuator model with a learned network mapping joint-command history to realized torque after identifying actuator error as the dominant transfer obstacle on ANYmal. The method attacks the dynamics component of the gap directly, where domain randomization only averages over it. and design optimization, is early. Johnny Nuñez Cano and his NVIDIA coauthors call Drake the "gold standard" for contact-implicit trajectory optimisation and rigorous numerics rather than throughput Cano 2026↓The State of Simulation for Physical AI: An OverviewJohnny Nuñez Cano, Mitesh Patel, Asier Arranz et al., 2026. This is the overview authors' positioning, not a comparative benchmark.