Reward Design and the MPC Debate

Reward design is writing the objective a reinforcement learning policy learns to maximize: in legged locomotion, a weighted sum of terms whose weights are tuned by hand^{Source 1}1 Rudin et al. 2021, Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Nikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021, and now also reward code written by language models^{Source 5}5 Ma et al. 2024, Eureka: Human-Level Reward Design via Coding Large Language Models. Yecheng Jason Ma, William Liang, Guanzhi Wang et al., ICLR 2024. Model-predictive control instead plans actions online with an explicit dynamics model^{Source 9}9 Zhang et al. 2026, Whole-Body Model-Predictive Control of Legged Robots with MuJoCo. John Z. Zhang, Taylor A. Howell, Zeji Yi et al., ICRA 2026.

Photo: Gabriel B. Margolis et al. / arXiv. Licence: CC BY 4.0.

The reward is the product

Rudin and colleagues define a weighted sum of nine reward terms, including feet air time, and prescribe no gait schedule^{Source 1}1 Rudin et al. 2021, Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Nikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021. The initial legged_gym commit (October 2021) declares fifteen entries in LeggedRobotCfg.rewards.scales. Nine are nonzero, from velocity tracking and joint torque to feet air time and action rate, and the joint-limit and contact-force functions among its nineteen reward functions are inactive in this config^{Source 2}2 Rudin 2021, Isaac Gym Environments for Legged Robots. Nikita Rudin, 2021.

Rudin and colleagues report artifacts such as a dragging leg and unreasonably high or low base heights, and tuned the weights before hardware transfer^{Source 1}1 Rudin et al. 2021, Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Nikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021. Margolis and Agrawal note that a policy failing in a new environment needs "a slow and iterative cycle of reward and environment redesign"^{Source 16}16 Margolis et al. 2022, Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior. Gabriel B. Margolis, Pulkit Agrawal, Conference on Robot Learning (CoRL) 2022. Rudin's PPOPPOPPO (proximal policy optimization) is the on-policy reinforcement learning algorithm the locomotion literature standardized on. Schulman and colleagues introduced it as a policy gradient method alternating between sampling environment data and optimizing a surrogate objective by stochastic gradient ascent. The surrogate allows several minibatch epochs per batch where standard policy gradient takes one step. It keeps most of trust region policy optimization's reliability without second-order machinery, and suits massively parallel simulation, whose thousands of robots supply its enormous on-policy batches. adapts its learning rate from the KL divergence between new and old policies: with a target of 0.01, it lowers the rate when KL exceeds twice the target and raises it under half^{Source 1}1 Rudin et al. 2021, Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Nikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021.

Three answers to weight tuning

The first answer adds constraints. Yunho Kim and colleagues use three rewards (command tracking, joint torque, action smoothness) and 11 constraints, six probabilistic and five average, training separate controllers in simulation for Raibo, Mini-cheetah, Hound, ANYmal B, ANYmal C, Unitree Go1 and Atlas and deploying on Raibo and Mini-cheetah. Across robots they adjusted only the torque coefficient among the rewards, found by trial and error, though constraint limits and PD gains still changed, and the probabilistic and average limits are not per-step hard bounds^{Source 6}6 Kim et al. 2024, Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotion. Yunho Kim, Hyunsik Oh, Jeonghyun Lee et al., IEEE Transactions on Robotics 2024.

Dohyeong Kim and colleagues split acrobatic tasks into manual stages with their own rewards and costs, five reward and five cost functions for a back-flip; stage-transition rules, cost thresholds, preferences and hyperparameters stay hand-set, and a real Go1 performs back-flips, side-rolls and two-hand walking^{Source 8}8 Kim et al. 2024, Stage-Wise Reward Shaping for Acrobatic Robots: A Constrained Multi-Objective Reinforcement Learning Approach. Dohyeong Kim, Hyeokjin Kwon, Junseok Kim et al., 2024.

The second answer adapts weights during training. Srisuchinnawong and Manoonpong's ROGER shifts gains between the primary reward and penalties near selected constraint thresholds, with penalties subtracted from the primary reward; the revised paper reports an exploration-induced violation in simulation and limits its safety claims to assumptions of zero-penalty optimality and gentle system and learning dynamics^{Source 7}7 Srisuchinnawong et al. 2025, Gain Tuning Is Not What You Need: Reward Gain Adaptation for Constrained Locomotion Learning. Arthicha Srisuchinnawong, Poramate Manoonpong, RSS 2025. The third answer removes per-skill tuning: BeyondMimic covers many agile humanoid skills with one tracking setup and shared hyperparameters^{Source 17}17 Liao et al. 2025, BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion. Qiayuan Liao, Takara E. Truong, Xiaoyu Huang et al., 2025, and language models can write the reward code itself.

Eureka: the reward as code under evolution

Eureka searches over reward code while PPO learns the policies. A language model receives a task description and environment observation code, with the existing reward excluded, and proposes Python reward functions; a separate task-fitness function selects candidates, and fitness and reward-component snapshots feed the next mutation^{Source 5}5 Ma et al. 2024, Eureka: Human-Level Reward Design via Coding Large Language Models. Yecheng Jason Ma, William Liang, Guanzhi Wang et al., ICLR 2024.

The evaluation covers 29 simulated tasks, and the headline is 83% of tasks and 52% average normalized improvement over each benchmark's original shaped reward; Eureka matches or beats the human reward on all nine Isaac tasks and 15 of 20 Dexterity tasks. Search reports the best reward across restarts, and final rewards are scored by the mean of maximum task fitness over ten checkpoints. Human-normalized scores are bounded to [0, 3] for the headline, so 52% measures normalized reward improvement^{Source 5}5 Ma et al. 2024, Eureka: Human-Level Reward Design via Coding Large Language Models. Yecheng Jason Ma, William Liang, Guanzhi Wang et al., ICLR 2024.

def reward(obs, act):
    # task: walk forward at 1.0 m/s
    return obs.base_lin_vel_x

Eureka needs a task-fitness function, which its authors say can diverge from human intent; they show an extension with human-written reward reflection^{Source 5}5 Ma et al. 2024, Eureka: Human-Level Reward Design via Coding Large Language Models. Yecheng Jason Ma, William Liang, Guanzhi Wang et al., ICLR 2024. Specification gaming satisfies the literal objective without achieving the intended outcome, and DeepMind researchers have collected around 60 examples^{Source 18}18 Krakovna et al. 2020, Specification gaming: the flip side of AI ingenuity. Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik et al., Google DeepMind blog, 2020. DrEureka writes both reward functions and domain-randomization distributions, getting a quadruped to walk atop a yoga ball^{Source 19}19 Ma et al. 2024, DrEureka: Language Model Guided Sim-To-Real Transfer. Yecheng Jason Ma, William Liang, Hung-Ju Wang et al., RSS 2024.

Curriculum: staying on the difficulty frontier

Reward shapingreward shapingReward shaping is adding rewards to a task's base reward to guide learning, usually densifying a sparse success signal. Ng, Harada, and Russell asked which changes leave a Markov decision process's optimal policy unchanged. They proved a shaping reward equal to a potential-function difference across the two states is sufficient, and effectively necessary. Classic shaping bugs, where the agent harvests the bonus instead of doing the task, come from non-potential-based rewards, which the theorem excludes. decides what to optimize; curriculumcurriculum learningCurriculum learning is training on tasks that start easy and grow harder as the agent improves. Rudin et al. adjusted each robot's terrain level by its progress. Their documented curriculum-trained policy for simulation and deployment used 4,096 robots, 98,304 transitions per batch and 1,500 updates in under twenty minutes on an i9-11900k CPU and RTX A6000 GPU. Their separate flat-terrain headline is under four minutes on one workstation GPU. decides in what order difficulty appears. Ng and colleagues showed that a shaping reward expressible as a difference of potentials leaves the optimal policy unchanged, and that this form is necessary for invariance^{Source 20}20 Ng et al. 1999, Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. Andrew Y. Ng, Daishi Harada, Stuart Russell, ICML 1999.

A terrain curriculum promotes robots that succeed on a tile to harder terrain and demotes those that fail^{Source 1}1 Rudin et al. 2021, Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Nikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021. Manipulation lacks such cheap evaluation, since its MDP is not cheaply simulatable. Automatic Domain Randomization widens the randomization distribution as success crosses a threshold, an idea from OpenAI's dexterous hand work that Isaac Lab ships as a configurable curriculum^{Source 4}4 OpenAI et al. 2019, Solving Rubik's Cube with a Robot Hand. OpenAI, Ilge Akkaya, Marcin Andrychowicz et al., 2019 ^{Source 3}3 NVIDIA et al. 2025, Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning. NVIDIA, Mayank Mittal, Yunrong Guo et al., 2025, and a velocity-command curriculum is one of the two main components of Margolis and colleagues' Mini Cheetah controller^{Source 21}21 Margolis et al. 2022, Rapid Locomotion via Reinforcement Learning. Gabriel B. Margolis, Ge Yang, Kartik Paigwar et al., Robotics: Science and Systems (RSS) 2022.

Paper figure: Viktor Makoviychuk et al. / arXiv. Licence: CC BY 4.0.

BVER, from Marco Hutter (ETH Zurich) and colleagues (October 2026), borrows from bidirectional RRT, growing start states outward from the goal and goals outward from the start; on quadruped box climbing it reached 95% success on a 0.4 m box in about 65% fewer iterations than the best reference-free curriculum it was compared with^{Source 32}32 Pfammatter et al. 2026, Bidirectional Voronoi-biased Exploration Curriculum for Reinforcement Learning. Juri Pfammatter, Kaixian Qu, Clemens Schwarke et al., 2026.

Success Guided Sampling (SGS), from Octi Zhang, Byron Boots and colleagues at the University of Washington and NVIDIA (CoRL 2026), applies the same idea to task configurations in general. It samples the configurations near the edge of what the policy can do, because uniform sampling spends a growing share of each batch on configurations already mastered or still out of reach. With up to about a million parallel environments, it solved multi-terrain quadruped locomotion and contact-rich assembly tasks that the prior methods it was compared with failed to solve, and the distilled RGB assembly policies transferred zero-shot to real hardware^{Source 34}34 Zhang et al. 2026, A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control. Octi Zhang, Mateo Guaman Castro, Patrick Yin et al., CoRL 2026.

The classical baseline got stronger

Classical legged control simplifies the model in its MPCmodel predictive controlModel predictive control is constant re-planning against an explicit dynamics model: each step it optimizes a short action sequence, executes only the first action and re-solves from the measured state. Per-step feedback corrects model error instead of freezing it into policy weights, so MPC shows less frozen-policy sim-to-real failure; a wrong model still costs closed-loop performance. Long assumed too slow for full robot dynamics, whole-body MPC now runs in real time on quadruped and humanoid hardware with Zhang and colleagues' MuJoCo-based iLQR controller. Posted in 2025, it was revised in 2026 to appear at ICRA 2026. layer. MIT Cheetah 3 cast its dynamics as a convex problem that still captures the full 3D system, planning ground reaction forces over horizons up to 0.5 s and solving in under 1 ms^{Source 22}22 Di Carlo et al. 2018, Dynamic Locomotion in the MIT Cheetah 3 Through Convex Model-Predictive Control. Jared Di Carlo, Patrick M. Wensing, Benjamin Katz et al., IEEE/RSJ IROS, 2018. Zhang and colleagues call this tradition the "model hierarchies commonly seen in traditional model-based MPC", built on "reduced-order models and hierarchical control approaches"^{Source 9}9 Zhang et al. 2026, Whole-Body Model-Predictive Control of Legged Robots with MuJoCo. John Z. Zhang, Taylor A. Howell, Zeji Yi et al., ICRA 2026.

Whole-body MPC itself ran on legged hardware earlier: Sleiman and colleagues' 2021 framework plans locomotion and manipulation as one multi-contact optimal control problem, solved in real time onboard, and pushed and pulled a heavy resistive door^{Source 33}33 Sleiman et al. 2021, A Unified MPC Framework for Whole-Body Dynamic Locomotion and Manipulation. Jean-Pierre Sleiman, Farbod Farshidian, Maria Vittoria Minniti et al., IEEE Robotics and Automation Letters, 2021. DIAL-MPC optimizes full-order quadruped dynamics in real time without training and outperforms RL policies by 50% on climbing tasks^{Source 23}23 Xue et al. 2024, Full-Order Sampling-Based MPC for Torque-Level Locomotion Control via Diffusion-Style Annealing. Haoru Xue, Chaoyi Pan, Zeji Yi et al., arXiv preprint (ICRA 2025), 2024.

Zhang and colleagues' MuJoCo-based iLQR controller, revised in 2026 to appear at ICRA 2026, runs on Unitree quadrupeds and an H1 humanoid with one warm-started iLQR update per solve. On a 20-core 13th-generation Intel i9, iLQR and TV-LQR feedback run at about 50 Hz and 300 Hz; the experiments fuse 100 Hz OptiTrack measurements with 500 Hz joint data, and onboard-only deployment is untested. Raising MuJoCo's impratio from 1 to 100 to reduce slipping doubled a quadruped iteration from 10 to 20 ms^{Source 9}9 Zhang et al. 2026, Whole-Body Model-Predictive Control of Legged Robots with MuJoCo. John Z. Zhang, Taylor A. Howell, Zeji Yi et al., ICRA 2026.

MPC re-solves against the measured state at every step, so feedback corrects part of the model mismatch online, though not all of it: Zhang and colleagues call for better system identification "to overcome the significant sim-to-real gap"^{Source 9}9 Zhang et al. 2026, Whole-Body Model-Predictive Control of Legged Robots with MuJoCo. John Z. Zhang, Taylor A. Howell, Zeji Yi et al., ICRA 2026, and Boston Dynamics says MPC works well when "the controller's model behaves similarly to the physical system". Its 2024 Spot account keeps the model-based locomotion controller while a learned policy removes the need to run multiple MPC instances in parallel^{Source 11}11 Boston Dynamics 2024, Starting on the Right Foot with Reinforcement Learning. Boston Dynamics, 2024, and in its August 2025 Atlas report with TRI, language-conditioned policies drive the robot through the control interface of an MPC-based teleoperation system^{Source 12}12 Cousineau et al. 2025, Large Behavior Models and Atlas Find New Footing. Eric Cousineau, Scott Kuindersma, Lucas Manuelli et al., 2025.

The case for each side

The case for learning: sim-trained controllers took two generations of ANYmal through natural environments "beyond the reach of prior published work in legged locomotion", in Lee and colleagues' words, at the cost of one forward pass at deploy time^{Source 10}10 Lee et al. 2020, Learning Quadrupedal Locomotion over Challenging Terrain. Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen et al., Science Robotics 5(47), 2020. Miller and colleagues report an end-to-end RL policy on Spot at over 5.2 m/s, more than triple the default controller's maximum speed^{Source 24}24 Miller et al. 2025, High-Performance Reinforcement Learning on Spot: Optimizing Simulation Parameters with Distributional Measures. AJ Miller, Fangzhou Yu, Michael Brauckmann et al., arXiv preprint, 2025. In drone racing, RL beat optimal control because it "optimizes a better objective" and copes with model uncertainty through domain randomizationdomain randomizationDomain randomization is training a policy across a distribution of simulators with randomized visual appearance, dynamics or both, so the real world reads as one more sample. Tobin et al. randomized rendering and transferred an object detector to a real arm. Peng et al. randomized dynamics and transferred a pushing policy with no real-world training. The cost is conservatism: such a policy is optimal for no single instance.^{Source 25}25 Song et al. 2023, Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning. Yunlong Song, Angel Romero, Matthias Mueller et al., Science Robotics 2023, adg1462, and semantic, language-conditioned control comes from the learned side, as Gemini Robotics 2 shows on humanoid whole-body control^{Source 13}13 Parada 2026, Gemini Robotics 2 brings whole body intelligence to robots. Carolina Parada, 2026.

The case for model-based control: a deliberately simple whole-body iLQR baseline runs in real time on quadruped and humanoid hardware with no task data^{Source 9}9 Zhang et al. 2026, Whole-Body Model-Predictive Control of Legged Robots with MuJoCo. John Z. Zhang, Taylor A. Howell, Zeji Yi et al., ICRA 2026. Grandia and colleagues optimize all of ANYmal's degrees of freedom in real time against foothold constraints from elevation maps^{Source 26}26 Grandia et al. 2022, Perceptive Locomotion through Nonlinear Model Predictive Control. Ruben Grandia, Fabian Jenelten, Shaohui Yang et al., 2022, and sampling MPC now discovers gaits and contact sequences without handcrafted patterns, in real time on a CPU^{Source 27}27 Schramm et al. 2025, Reference-Free Sampling-Based Model Predictive Control. Fabian Schramm, Pierre Fabre, Nicolas Perrin-Gilbert et al., arXiv preprint (ICRA 2026), 2025. Convex MPC can enforce friction-cone limits as hard constraints, as MIT Cheetah 3's planner does, where a reward penalty only discourages violations^{Source 22}22 Di Carlo et al. 2018, Dynamic Locomotion in the MIT Cheetah 3 Through Convex Model-Predictive Control. Jared Di Carlo, Patrick M. Wensing, Benjamin Katz et al., IEEE/RSJ IROS, 2018.

Newton's hydroelastic contact modeling is inspired by Drake's, Drake developer TRI partners with Newton on solvers and contact, and Newton is now a physics backend for Isaac Lab^{Source 14}14 Nuñez Cano et al. 2026, The State of Simulation for Physical AI: An Overview. Johnny Nuñez Cano, Mitesh Patel, Asier Arranz et al., 2026 ^{Source 15}15 Reist et al. 2026, Newton Adds Contact-Rich Manipulation and Locomotion Capabilities for Industrial Robotics. Philipp Reist, Miguel Zamora Mora, JC Chang et al., 2026. Massively parallel sampling MPC on a real Franka still reports contact sensitivity as a main challenge^{Source 28}28 Dierking et al. 2026, Real-World Deployment of Massively Parallel Sampling-Based MPC for Contact-Rich Manipulation. Magnus Dierking, Joao Carvalho, An Thai Le et al., ICRA 2026 Workshop on Frontiers of Optimization for Robotics.

Where the debate landed

Hybrids now combine the two: MPC-guided policy search used MPC with full state to generate training data for a simulated quadrotor's network, which runs at a fraction of MPC's cost^{Source 29}29 Zhang et al. 2015, Learning Deep Control Policies for Autonomous Aerial Vehicles with MPC-Guided Policy Search. Tianhao Zhang, Gregory Kahn, Sergey Levine et al., 2015. DTC trains an RL policy to track footholds from a model-based planner and reports more robustness than model-based counterparts on slippery or deformable ground^{Source 30}30 Jenelten et al. 2023, DTC: Deep Tracking Control. Fabian Jenelten, Junzhe He, Farbod Farshidian et al., 2023, and a policy trained entirely in a differentiable simulator transferred zero-shot to a real quadruped^{Source 31}31 Schwarke et al. 2024, Learning Deployable Locomotion Control via Differentiable Simulation. Clemens Schwarke, Victor Klemm, Joshua Bagajo et al., arXiv preprint (CoRL 2025), 2024.