Reward Design and the MPC Debate

Every module so far in this domain assumed someone had already written the objective. The parallel-training module showed thousands of robots collecting reward in lockstep; the sim-to-real module showed how to make the resulting policy survive contact with hardware. This module is about where the reward comes from, and it closes with the argument the whole domain has been circling: whether you need a learned policy at all, when classical trajectory optimization keeps taking back ground.

Twelve weighted terms sit on the illustrative behavior preview below. Its weights, per-term magnitudes and freeze/prance/chatter categories are teaching choices, not the paper's nine-term objective or a runnable legged_gym configuration. Drag the controls to explore this toy weighted sum.

This local teaching model uses twelve authored terms, weights and fixed per-term magnitudes. Its signed weighted total is dimensionless per step, not a measured reward. Freeze, prance and chatter are chosen classification rules and drawn poses, not trained policies, measured control frequencies or actuator-damage predictions.

The reward is the product

Rudin and colleagues define a weighted sum of nine reward terms in the paper, including feet air time; gait-independent means that the reward does not prescribe a gait schedule, not that air time is absent Rudin 2021↓Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement LearningNikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021. The separate legged_gym initial commit (October 2021, pinned by commit hash in the cited record's audit trail) declares fifteen entries in LeggedRobotCfg.rewards.scales, of which nine are nonzero: linear- and angular-velocity tracking, vertical linear velocity, roll/pitch angular velocity, joint torque, joint acceleration, feet air time, collisions, and action rate. _prepare_reward_function removes zero scales and multiplies the remaining scales by the environment time step. The base class implements nineteen reward functions, including joint-limit and contact-force functions, but those definitions do not make them active terms in this config. Its zero-valued feet_stumble entry does not even match the _reward_stumble function name. That function penalizes feet hitting vertical surfaces, not foot slip; the air-time function rewards long steps on first contact and requires a planar-command norm above 0.1. These are properties of this pinned base code, not a universal or production reward recipe Rudin 2021↓Isaac Gym Environments for Legged RobotsNikita Rudin, 2021.

The preview maps weight settings to three illustrative categories: freeze, prance, and chatter. Those classifications and the displayed motion are outputs of this local teaching model, not policies trained in the cited simulator. Rudin and colleagues report different observed artifacts, including a dragging leg and unusually high or low base height, and describe tuning reward weights before transferring their policy to hardware Rudin 2021↓Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement LearningNikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021.

Rudin and colleagues use an adaptive learning rate in their PPOPPOProximal policy optimization, the on-policy reinforcement learning algorithm the locomotion literature standardized on. Schulman and colleagues introduced it as a policy gradient method that alternates between sampling data through interaction with the environment and optimizing a surrogate objective with stochastic gradient ascent, where the surrogate is what allows several epochs of minibatch updates on each batch of samples instead of the single gradient step standard policy gradient takes. It keeps most of trust region policy optimization's reliability without the second-order machinery, and it pairs naturally with massively parallel simulation, where thousands of robots supply the enormous on-policy batches it consumes. implementation, based on the KL divergence between the updated and previous policies. Table 3 gives a desired KL of 0.01; Algorithm 1 lowers the learning rate when KL exceeds twice the target and raises it when KL is below half the target, within stated learning-rate limits Rudin 2021↓Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement LearningNikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021. This mechanism does not establish that reward retuning is stable.

Three answers to weight tuning

The field has responded to the tuning burden along three distinct lines, and they should not be conflated.

The first approach reduces reward tuning by adding explicit constraints. Yunho Kim and colleagues use three reward terms for command tracking, joint torque, and action smoothness, alongside 11 constraints: six probabilistic and five average constraints. They trained separate controllers in simulation for Raibo, Mini-cheetah, Hound, ANYmal B, ANYmal C, Unitree Go1, and Atlas; hardware deployment was on Raibo and Mini-cheetah. For transfer between robots, they adjusted only the torque-reward coefficient among the three reward coefficients, after choosing a reference-robot coefficient by trial and error. Constraint limits and other robot-specific settings still changed. Their IPO-based optimizer uses TRPO and adaptive constraint thresholds; these probabilistic and average limits are not hard bounds enforced at every training step Kim 2024↓Not Only Rewards But Also Constraints: Applications on Legged Robot LocomotionYunho Kim, Hyunsik Oh, Jeonghyun Lee et al., IEEE Transactions on Robotics 2024.

Dohyeong Kim and colleagues take a different constrained multi-objective approach to acrobatics. They manually divide tasks into stages and define separate reward and cost functions for each stage. Their back-flip example has five stages, Stand-Sit-Jump-Air-Land, with five reward functions and five cost functions. CoMOPPO combines normalized reward and cost advantages in a PPO update; stage-transition rules, cost thresholds, objective preferences, and optimization hyperparameters still need to be specified. The reported simulation tasks use Unitree Go1 and H1; the real-world demonstrations use Go1 for back-flips, side-rolls, and two-hand walking Kim 2024↓Stage-Wise Reward Shaping for Acrobatic Robots: A Constrained Multi-Objective Reinforcement Learning ApproachDohyeong Kim, Hyeokjin Kwon, Junseok Kim et al., 2024.

The second approach adapts reward weights during training. Srisuchinnawong and Manoonpong’s ROGER uses estimates of nonnegative penalty magnitudes relative to selected constraint thresholds to reduce the primary-reward gain and increase penalty gains near the thresholds, reversing that balance farther away. The penalties are subtracted from the primary reward, not added as bonuses. This is not a universal constraint-satisfaction guarantee: the revised paper reports an exploration-induced violation in its simulated quadruped experiment and explicitly limits its safety claims to assumptions about zero-penalty optimality and gentle system and learning dynamics Srisuchinnawong 2025↓Gain Tuning Is Not What You Need: Reward Gain Adaptation for Constrained Locomotion LearningArthicha Srisuchinnawong, Poramate Manoonpong, RSS 2025.

The third outsources the writing to a language model. That one deserves its own section.

Eureka: the reward as code under evolution

Eureka searches over reward code; PPO learns the policies. In the default experiment, GPT-4 receives a task description and environment observation code with the existing reward excluded, then proposes Python reward functions. Candidate rewards train PPO policies in Isaac Gym. Selection uses a separate task-fitness function, while scalar snapshots of that fitness and each reward component provide feedback for the next reward-code mutation. The implementation trims the context to the relevant observation code and the last reward/reflection pair.
Source: Ma 2024↓Eureka: Human-Level Reward Design via Coding Large Language ModelsYecheng Jason Ma, William Liang, Guanzhi Wang et al., ICLR 2024.

The evaluation covers 29 simulated tasks: nine original Isaac Gym environments and 20 Dexterity tasks, spanning 10 robot morphologies. The paper's headline reports 83% of tasks and 52% average normalized improvement over human-expert rewards; its detailed results describe matching or exceeding the human reward on all nine Isaac tasks and 15 of the 20 Dexterity tasks, not strict wins everywhere. The human baseline is each benchmark's original shaped reward Ma 2024↓Eureka: Human-Level Reward Design via Coding Large Language ModelsYecheng Jason Ma, William Liang, Guanzhi Wang et al., ICLR 2024.

Search uses five independent runs, five iterations per run and 16 candidate rewards per iteration. The reported comparison uses the best reward across search restarts; the separate reward-reflection ablation instead averages restarts. Intermediate rewards receive one PPO run; final rewards receive five, scored by the average of the maximum task fitness over ten fixed-interval checkpoints Ma 2024↓Eureka: Human-Level Reward Design via Coding Large Language ModelsYecheng Jason Ma, William Liang, Guanzhi Wang et al., ICLR 2024.

The human-normalized score is (Method - Sparse) / abs(Human - Sparse); for the headline average across all 29 tasks, Appendix D adjusts each score to lie in [0, 3]. The 52% headline is not a 52-percentage-point increase in success rate Ma 2024↓Eureka: Human-Level Reward Design via Coding Large Language ModelsYecheng Jason Ma, William Liang, Guanzhi Wang et al., ICLR 2024.

The panel below is a scripted teaching example of the Eureka loop, not a recorded Eureka or PPO experiment. All three generations, reward code, statistics, fitness scores and reflections are authored fixtures. The sequence illustrates sprinting and falling, standing still, then tracking a command; clicking advances the script without training a policy or calling a language model.

def reward(obs, act):
    # task: walk forward at 1.0 m/s
    return obs.base_lin_vel_x

The limitations are structural, and the 2026 literature treats them as open. Eureka needs a fitness function, which is itself a reward-specification problem one level up. It needs cheap parallel training per candidate, so it inherits the locomotion-manipulation asymmetry from the first module in this domain. And LLM-written rewards are prone to specification gaming that only shows up on hardware; nobody has published a systematic study of whether they induce more or less contact-solver exploitation than human-written ones.

Lee and colleagues' Reward Design Agent (RDA) compares against a matched Eureka implementation with the same GPT-5 backbone, task instructions, and training budget. That baseline reflects on numerical reward-component statistics; RDA additionally analyzes rendered trajectories together with reward logs, scores subtasks, summarizes failures, and revises both the subtask descriptions and reward code. Its experiments cover 12 ManiSkill tabletop tasks and four HumanoidBench whole-body tasks, using SAC with SimbaV2. On HumanoidBench, the paper reports higher VLM-rated alignment for RDA (0.70 versus Eureka's 0.47), with RDA success at 0.42 and slightly below Eureka's: reaching the goal and following the intended behavior are different measurements. The package example illustrates the distinction, contrasting throwing or torso-pushing with the requested hand-contact pushing. Reported alignment comes from GPT-4.1 ratings of five rollout videos, each queried four times, not a guarantee of instruction correctness; the authors identify costly RL training, imperfect fine-grained visual reasoning, and conflicting catch-versus-throw rewards in basketball as limitations Lee 2026↓RDA: Reward Design Agent for Reinforcement LearningHojoon Lee, Ajay Subramanian, Ben Abbatematteo et al., RLC 2026. Eureka itself already noted that task fitness need not match human intent and demonstrated an extension using human-written reward reflection, so the comparison concerns RDA's automated visual diagnostics, not the discovery of a previously unacknowledged limitation Ma 2024↓Eureka: Human-Level Reward Design via Coding Large Language ModelsYecheng Jason Ma, William Liang, Guanzhi Wang et al., ICLR 2024.

Curriculum: staying on the difficulty frontier

Reward shapingreward shapingSupplying extra training rewards on top of a task's base reward to guide the learning agent, usually to turn a sparse success signal into a denser one that learning can climb. Ng, Harada, and Russell asked exactly which modifications to a Markov decision process's reward function leave the optimal policy unchanged, and proved that a transition reward expressible as the difference of a potential function across the two states is sufficient, and effectively necessary, for that invariance. The theorem explains the classic shaping bugs, where the agent learns to harvest the bonus instead of doing the task: those bugs come from non-potential-based rewards, the kind the theorem rules out. decides what to optimize; curriculumcurriculum learningTraining on a scheduled sequence of tasks that grow harder as the agent improves, instead of sampling the full difficulty range from the start. Rudin and colleagues raised or lowered terrain levels based on each robot's progress. Their documented curriculum-trained simulation-and-deployment policy used 4,096 robots, 98,304 transitions per batch and 1,500 updates in under twenty minutes on an i9-11900k CPU and RTX A6000 GPU; their separate flat-terrain headline is under four minutes on one workstation GPU. decides in what order the policy is allowed to see difficulty. Three mechanisms dominate, and they are different enough to keep separate.

Terrain curriculum is the game-inspired scheme from the massively parallel walking work: robots that succeed on a terrain tile get promoted to a harder one, robots that fail get demoted, and thousands of parallel robots make the difficulty frontier measurable rather than guessed Rudin 2021↓Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement LearningNikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021. Automatic Domain Randomization widens the randomization distribution itself as success crosses a threshold, an idea from OpenAI's dexterous hand work that now ships in Isaac Lab as a configurable curriculum OpenAI 2019↓Solving Rubik's Cube with a Robot HandOpenAI, Ilge Akkaya, Marcin Andrychowicz et al., 2019 NVIDIA 2025↓Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot LearningNVIDIA, Mayank Mittal, Yunrong Guo et al., 2025.

For population-based training, Isaac Lab v1 describes DexPBT workers running independent RL processes with their own hyperparameters; strong workers replace weak workers’ weights and hyperparameters. Its RL-Games implementation reproduces a 6D reposing task using eight workers with one to two GPUs each, converging in approximately 16 hours on NVIDIA OVX L40 hardware
Source: NVIDIA 2025↓Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot LearningNVIDIA, Mayank Mittal, Yunrong Guo et al., 2025.

Notice what all three assume: cheap, dense, parallel evaluation. Curriculum is a dividend of the throughput covered in the parallel-sim module, and it is one more reason the same machinery does not transfer to manipulation, where the MDP is not cheaply simulatable.

The classical baseline got stronger

Everything above is about making learning work. The learning-vs-model-based debate asks whether it had to. The classical stack for legged control is a hierarchy: a footstep and contact planner, an MPCmodel predictive controlControl by constant re-planning: at each control step, optimize a short sequence of future actions against an explicit dynamics model, execute only the first action, and solve again from the freshly measured state. Because the plan is recomputed online, model error is rejected by feedback at every step instead of being frozen into a policy's weights, so a model-predictive controller shows less of the frozen-policy sim-to-real failure mode, though a wrong dynamics model still costs it closed-loop performance. Long assumed too slow for a robot's full dynamics, the method reached whole-body scale in 2026, when Zhang and colleagues solved iLQR against MuJoCo's physics fast enough to close the loop in real time on real hardware. layer optimizing a reduced-order model, and a whole-body QP tracking the MPC output against full rigid-body dynamics, friction cones, and actuator limits. The reduced-order model in the middle layer existed because full-order MPC was assumed too slow to run online. Whole-body MPC work describes the tradition it replaces as the "model hierarchies commonly seen in traditional model-based MPC", built on "reduced-order models and hierarchical control approaches" Zhang 2026↓Whole-Body Model-Predictive Control of Legged Robots with MuJoCoJohn Z. Zhang, Taylor A. Howell, Zeji Yi et al., ICRA 2026.

Zhang and colleagues' March 2026 revision, listed on arXiv as "to appear at ICRA 2026", demonstrates a MuJoCo-based iLQR controller on Unitree Go1 and Go2 quadrupeds, a Go1 walking on its hind legs and transitioning to a handstand, and an H1 humanoid tracking a trotting gait. It uses forward finite differences for dynamics and residual Jacobians, analytical norm derivatives, approximate cost Hessians, and one warm-started iLQR update per solve without waiting for convergence. Joint-angle targets feed low-level PD control, which is also included in the planning model. On a desktop with a 20-core, 13th-generation Intel i9, the reported iLQR and intervening TV-LQR feedback rates are approximately 50 Hz and 300 Hz; the default horizon is 0.35 s with 100-Hz model discretization. H1 uses a 0.5-s horizon, ten active joint actuators, and four foot contact points after disabling upper-body DoFs with unreliable relative encoders and collision checking elsewhere on the body. These are laboratory experiments using 100-Hz OptiTrack measurements fused with 500-Hz joint measurements, not an onboard-only deployment. The authors' phrase "few sim-to-real considerations" includes consequential setup choices: increasing MuJoCo's impratio from 1 to 100 to reduce slipping roughly doubled a quadruped iteration from 10 to 20 ms on a separate 12th-generation i7 test. An eight-second H1 feedback comparison also records a gantry-assisted recovery in the trial without TV-LQR feedback. The paper presents an accessible model-based baseline, while identifying motion-capture dependence, actuation/contact system identification, contact-mode exploration, and long-horizon numerical behavior as limitations. It does not establish calibration-free operation, zero total engineering cost, or blanket superiority over learned control Zhang 2026↓Whole-Body Model-Predictive Control of Legged Robots with MuJoCoJohn Z. Zhang, Taylor A. Howell, Zeji Yi et al., ICRA 2026.

The structural point is deeper than one paper. MPC has no sim-to-real gap in the RL sense: it re-solves against the measured state at every control step, so model error is rejected by feedback instead of being baked into weights. From this angle, domain randomizationdomain randomizationTraining a policy across a distribution of simulators with randomized parameters, visual appearance, dynamics, or both, so the real world reads as one more sample from that distribution. Tobin et al. randomized rendering and transferred an object detector to a real arm; Peng et al. randomized dynamics and transferred a pushing policy with no real-world training. The price is conservatism: a policy that does well across a wide distribution is optimal for no single instance of it. is an expensive workaround for not closing the loop on the model. Boston Dynamics describes a source-specific hybrid arrangement, rather than a choice between learning and models: its 2024 Spot account retains the model-based locomotion controller while a learned policy removes the need to evaluate multiple MPC instances in parallel Dynamics 2024↓Starting on the Right Foot with Reinforcement LearningBoston Dynamics, 2024. Its August 2025 Atlas/TRI report concerns language-conditioned policies for whole-body manipulation, including locomotion; the teleoperation system is built on MPC, and the learned policies use that same robot control interface Cousineau 2025↓Large Behavior Models and Atlas Find New FootingEric Cousineau, Scott Kuindersma, Lucas Manuelli et al., 2025. These two reports describe different robots and policy generations, not replacement of either robot's entire controls stack.

The debate, steelmanned

The honest comparison runs on the axes where the two stacks actually differ. The table states each; the interactive below it shows what the difference looks like when the ground stops being what either controller expected.

The case for learning, at its strongest: locomotion is settled, and sim-trained controllers handle terrain, unknown payloads, and hardware variation, taking two generations of ANYmal through ground beyond the reach of prior published work in legged locomotion (in the papers' own words), at the cost of one forward pass at deploy time Lee 2020↓Learning Quadrupedal Locomotion over Challenging TerrainJoonho Lee, Jemin Hwangbo, Lorenz Wellhausen et al., Science Robotics 5(47), 2020 Miki 2022↓Learning robust perceptive locomotion for quadrupedal robots in the wildTakahiro Miki, Joonho Lee, Jemin Hwangbo et al., Science Robotics 7(62), 2022. Perception integrates naturally; there is no clean way to put an RGB image into a QP. Contact schedules emerge from exploration instead of being enumerated. And semantics exists only on the learned side: Gemini Robotics 2 controls three embodiments from one checkpoint, and no model-based stack does anything comparable Parada 2026↓Gemini Robotics 2 brings whole body intelligence to robotsCarolina Parada, 2026.

The case for model-based control, at its strongest: the strongest single 2026 result in legged control is not a learning result Zhang 2026↓Whole-Body Model-Predictive Control of Legged Robots with MuJoCoJohn Z. Zhang, Taylor A. Howell, Zeji Yi et al., ICRA 2026. Constraints are enforced, not penalized; a friction cone in a QP holds, a reward penalty merely advises. Data efficiency is infinite in the relevant sense, since MPC needs zero task data. The behavior is verifiable in ways a network is not. And contact fidelity is a modeling problem, which is why the learning stack is importing model-based contact research into its simulators: Newton's hydroelastic contact modeling is inspired by Drake's contact model, Drake-developer TRI now partners with Newton on solver and contact modeling, and Newton acts as the physics layer under Isaac Lab Cano 2026↓The State of Simulation for Physical AI: An OverviewJohnny Nuñez Cano, Mitesh Patel, Asier Arranz et al., 2026 Reist 2026↓Newton Adds Contact-Rich Manipulation and Locomotion Capabilities for Industrial RoboticsPhilipp Reist, Miguel Zamora Mora, JC Chang et al., 2026. The learning stack's current progress on manipulation is gated by better models, not better algorithms.

Where the debate landed

The field has mostly stopped arguing and hybridized along a clean division. Model-based control owns anything with a good analytic model, hard constraints, and a need for guarantees: whole-body torque control, force control, safety layers, and increasingly the low-level interface below a learned policy. Learning owns anything requiring perception, semantics, contact discovery, or generalization across embodiments. The newest position dissolves the boundary entirely: model-based methods as the training signal for learned ones, through differentiable simulation, MPC-as-teacher distillation, and learned models used as forward dynamics for planning.

The genuinely contested question in 2026 is not learning or models. It is where the learned model lives: in the weights of a policy, in a latent dynamics model, in a video generator, or nowhere at all because you re-solve online.