Humanoid Whole-Body Control

The previous module covered how learned policies took over legged locomotion. This one covers the harder problem those same groups attacked next: making a full humanoid, legs through torso through arms through hands, do useful work without falling. The organizing idea across nearly all of it is motion tracking. Treat whole-body controlwhole-body controlTreating all of a robot's actuated degrees of freedom, legs, torso, arms, and hands, as one coupled control problem instead of stacking an arm controller on top of a separate locomotion controller. The model-based version optimizes against the full rigid-body dynamics at once: Zhang and colleagues ran whole-body model-predictive control in real time on hardware with plain iLQR and MuJoCo dynamics, across dynamic quadruped locomotion and full-sized humanoid walking. The learned version reframes the same problem as tracking a retargeted human motion, the route H2O took for real-time whole-body teleoperation, with reinforcement learning supplying the balance and contact feasibility that raw retargeting loses. as following a reference human motion, retargetedretargetingMapping motion recorded on one body, usually a human's, onto a robot with different proportions and joint limits, so human demonstrations become references the robot can track. Raw retargeting ignores the robot's physical constraints and can produce references that are morphologically infeasible, which is why systems such as H2O and ASAP pair retargeted human motion with reinforcement learning that restores balance and contact feasibility on the real body. onto the robot's kinematics, and use reinforcement learning to learn tracking under the robot's dynamics and hardware constraints. Retargeted human motion can be infeasible for the robot He 2024↓Learning Human-to-Humanoid Real-Time Whole-Body TeleoperationTairan He, Zhengyi Luo, Wenli Xiao et al., IROS 2024. The human demonstration carries the intent; the policy solves balance, contact timing, and actuator limits.

Where task-level intelligence plugs into that is unsettled, and the three answers shipping in 2026 are below. Switch between motion-tracking RL, the latent-action hierarchy, and end-to-end VLA, and watch the learning boundary move: each program drew it somewhere different.

All four figures are vendor-reported: the Helix 02 numbers from Figure's announcement Figure AI 2026↓Introducing Helix 02: Full-Body AutonomyFigure AI, 2026 and the Gemini Robotics 2 numbers from DeepMind's announcement Parada 2026↓Gemini Robotics 2 brings whole body intelligence to robotsCarolina Parada, 2026.

The tracking lineage

PHC (Perpetual Humanoid Control) demonstrated motion tracking and fall recovery for simulated avatars, not a physical robot. Its rotation-based controller achieved 98.9% imitation success on 11,313 filtered AMASS training clips; that is a training-set result, not universal tracking success. Imitation counts as a failure if mean body-joint deviation from the reference exceeds 0.5 m at any point. PMCP freezes learned motion primitives, trains new primitives on progressively harder failed subsets, adds a separate recovery primitive, and learns a multiplicative composer. The result is one composite controller rather than a separately trained controller for every clip. The paper demonstrates reset-free avatar control with webcam pose estimates and tracking of language-generated motion, but highly dynamic clips, noisy real-time estimates, and recovery artifacts remain limitations Luo 2023↓Perpetual Humanoid Control for Real-time Simulated AvatarsZhengyi Luo, Jinkun Cao, Alexander Winkler et al., ICCV 2023.

H2O adapts ideas from simulated humanoid animation to the Unitree H1 rather than transferring PHC unchanged. Its paper calls the privileged motion-filtering imitator "similar to PHC" and follows PULSE for that imitator's state space and training procedure. A separate policy learns whole-body tracking from the cleaned retargeted motion set with domain randomization, then follows human keypoints estimated in real time from an RGB camera. The deployed teleoperationteleoperationDirect remote control of a robot by a human operator, used in robot learning mostly to record expert demonstrations. The ALOHA rig made the setup cheap and precise: the operator back-drives a leader arm that is a joint-for-joint twin of the follower the policy will run on, so the recorded motion is already in the robot's own action space. Mobile ALOHA extended the same leader-follower scheme to a mobile base for whole-body tasks. system uses robot-side motion capture for linear velocity and a PD controller below the 19-dimensional joint-target policy He 2024↓Learning Human-to-Humanoid Real-Time Whole-Body TeleoperationTairan He, Zhengyi Luo, Wenli Xiao et al., IROS 2024.

OmniH2O uses kinematic motion goals as a shared interface for its Unitree H1 system. A deployable student is distilled from a privileged RL teacher and tracks head-and-hand position goals using proprioceptive history, without an explicit global-linear-velocity input. Robot root odometry is still needed to transform goals into the robot frame. The method defines target joint angles followed by PD control, while Appendix A describes torque outputs; the paper leaves that action-interface wording inconsistent He 2024↓OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and LearningTairan He, Zhengyi Luo, Xialin He et al., CoRL 2024.

VR and RGB pose input use the same tracker; language instructions first pass through a text-to-motion model. In the demonstrated GPT-4o setup, robot-camera images drive selection among motion primitives because of the model’s response time, not direct low-level motor control. For learning from teleoperated demonstrations, a visual policy predicts motion-goal sequences and hand commands, with the tracking policy underneath. The imitation-learning evaluation covers four of the six recorded tasks, with ten runs per task. These examples do not establish unrestricted task generalization, and the authors report no safety guarantees for extreme disturbances or out-of-distribution motion goals He 2024↓OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and LearningTairan He, Zhengyi Luo, Xialin He et al., CoRL 2024.

HumanPlus uses a single RGB camera to estimate the human operator’s body and hand poses, not as the robot’s only sensor. Its Humanoid Shadowing Transformer is trained with PPO on retargeted human motion in simulation and deployed zero-shot on a customized Unitree H1. The robot supplies IMU and joint-encoder observations; the policy outputs body-joint position setpoints for PD control. Hand targets pass directly to the controller, and seated operation bypasses the learned body controller. Operators collect demonstrations through shadowing while two head-mounted RGB cameras record the robot’s view. Supervised behavior cloningbehavior cloningTraining a policy by supervised learning on expert demonstrations: each recorded observation is an input, the action the expert took at that moment is the label, and the fitted mapping from state to action is the policy. The recipe predates deep learning; ALVINN (1988) used camera and laser-range inputs and trained on simulated road images before NAVLAB tests, and the same approach, scaled to modern networks and datasets, still underlies most learned manipulation policies. Its known failure mode in closed loop is covariate shift. then trains a separate Humanoid Imitation Transformer from binocular images and proprioception to supply target poses to the low-level controller. The paper demonstrates task-specific autonomous skills, not unrestricted skill generalization; occlusion, fixed retargeting, and limited locomotion scope remain constraints Fu 2024↓HumanPlus: Humanoid Shadowing and Imitation from HumansZipeng Fu, Qingqing Zhao, Qi Wu et al., 2024.

ExBody2 reports that drifting global keypoint targets can accumulate tracking errors. It uses local keypoint targets with separate velocity tracking. An initial policy ranks reference motions by lower-body tracking error for dataset filtering; a privileged PPO teacher is distilled into a student using observation history. The authors report walking, squatting, and dancing on Unitree G1. Their generalist and fine-tuned specialist policies trade broad coverage against motion-specific precision, and the paper leaves transitions between specialists unresolved Ji 2024↓ExBody2: Advanced Expressive Humanoid Whole-Body ControlMazeyu Ji, Xuanbin Peng, Fangchen Liu et al., 2024. ASAP then attacked the residual dynamics gap with the delta-action stage covered in the sim-to-real module: train the tracking policy in sim, roll it out on the real Unitree G1, learn a correction model from the mismatch, and fine-tune against the corrected simulator. That produced jumps and sports-style motions on hardware He 2025↓ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body SkillsTairan He, Jiawei Gao, Wenli Xiao et al., RSS 2025.

The 2025 to 2026 work consolidated. KungfuBot trains a separate policy for each processed reference motion rather than one policy for the whole repertoire. Its adaptive tracking rule tightens the exponential reward’s error tolerance from an exponential moving average of tracking error, with a non-increasing update. Separate curricula lower the early-termination threshold and increase regularization penalties. The authors demonstrate dynamic motions on Unitree G1; their quantitative hardware comparison covers ten Tai Chi trials with the robot root fixed to the origin for evaluation Xie 2025↓KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic SkillsWeiji Xie, Jinrui Han, Jiakun Zheng et al., NeurIPS 2025. GMT trains a unified motion-tracking controller rather than merging already-trained per-skill policies. Adaptive sampling re-clips long motions and adjusts sampling probabilities using completion and tracking errors. Its privileged teacher uses a learned soft mixture-of-experts: a gating network combines expert action outputs, and a deployable student learns from the teacher through DAgger. The authors demonstrate tracking on Unitree G1, but report the baseline comparisons and ablations in simulation. The controller does not support getting up after a fall or rolling, and is not designed for tracking on slopes and stairs Chen 2025↓GMT: General Motion Tracking for Humanoid Whole-Body ControlZixuan Chen, Mazeyu Ji, Xuxin Cheng et al., 2025. The 2026 robustness work targets what retargeting breaks: references that arrive noisy or locally infeasible in the robot's morphology get selectively aggregated by a multi-head cross-attention command encoder inside a dynamics-conditioned command aggregation framework, with a fall-recovery curriculum, so the controller survives reference defects instead of amplifying them Ma 2026↓Robust and Generalized Humanoid Motion TrackingYubiao Ma, Han Yu, Jiayin Xie et al., 2026.

Three decompositions ship in 2026

The lineage above answers how to make a humanoid move like a reference. It does not answer where task-level intelligence plugs in, which is the split drawn in the figure at the top of this module. All three frontier humanoid programs shipped whole-body control in 2026, and each drew the learning boundary in a different place.

A unified motion-tracking policy: Figure Helix 02 S0

Figure kept the tracking decomposition and industrialized it. Helix 02 is a three-layer stack: S2 does semantic reasoning over scenes and language and emits latent goals; S1, a visuomotor transformer at 200 Hz, takes every sensor in (head cameras, palm cameras, fingertip tactile, full proprioceptionproprioceptionSensing of the body's own state: joint positions and velocities, orientation, and contact, as opposed to exteroception, which senses the outside world through cameras or lidar. Proprioception is fast, cheap, and never occluded, and the ETH Zurich line showed how far it goes: Lee and colleagues trained an ANYmal controller that hiked mud, snow, rubble, and vegetation on proprioception alone, and Miki and colleagues kept it as the trusted channel the policy falls back on when its terrain map disagrees with its body.) and emits complete joint targets for the whole robot; S0, a 10M-parameter whole-body controller at 1 kHz, converts joint state and base motion into actuator commands. S0 was trained on over 1,000 hours of joint-level retargeted human motion plus sim-to-real RL across more than 200,000 parallel simulation environments, and Figure states it replaced 109,504 lines of hand-engineered C++ Figure AI 2026↓Introducing Helix 02: Full-Body AutonomyFigure AI, 2026. The architectural bet is the lineage's: learn one prior over how humans move while balancing, so the layers above never reason about footsteps. The headline demo, a 4-minute dishwasher unload of 61 loco-manipulation actions, is vendor-reported with no paper and no success rates.

A VLA above a learned controller: GR00T + GEAR-SONIC

NVIDIA kept the VLA and the whole-body controller as separate learned systems with a narrow interface between them. GR00T N1.7, a 3B-parameter VLA with a Cosmos-Reason2-2B backbone and a flow-matchingflow matchingA generative modeling recipe that learns a time-dependent vector field carrying samples from a noise distribution to the data distribution, trained by regressing the model's field against the conditional flow rather than by estimating a score. pi0 brought it to robot control: a dedicated action expert is trained with the flow-matching objective to denoise continuous action chunks, and at inference the learned field is integrated as an ODE for a handful of steps, which keeps the policy fast enough for high-frequency control while keeping the multimodality that made diffusion policies attractive. action head, predicts compact latent action tokens; the GEAR-SONIC whole-body controller, exposed as the UNITREE_G1_SONIC embodiment, decodes those tokens into full-body joint commands for legs, arms, and hands NVIDIA 2026↓NVIDIA Isaac GR00TNVIDIA, 2026 NVIDIA 2026↓GR00T-WholeBodyControl (GEAR-SONIC whole-body controller workflow)NVIDIA, 2026. One language-conditioned policy thereby produces coordinated locomotion and manipulation without the VLA ever naming a joint. The academic versions of the same decomposition are explicit about the interface: LeVERB learns a latent vision-language verb vocabulary from rendered kinematic demonstrations and hands it to an RL whole-body controller Xue 2025↓LeVERB: Humanoid Whole-Body Control with Latent Vision-Language InstructionHaoru Xue, Xiaoyu Huang, Dantong Niu et al., 2025, and WholeBodyVLA learns latent actions from action-free egocentric video for loco-manipulation Jiang 2025↓WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation ControlHaoran Jiang, Jin Chen, Qingwen Bu et al., ICLR 2026, 2025. GR00T's bet is that a constrained interface is a feature: the VLA can be retrained or swapped without touching the controller, and the controller's dynamics stay out of the VLA's context. This is also the only one of the three stacks with public code (Apache-2.0) and released weights NVIDIA 2026↓NVIDIA Isaac GR00TNVIDIA, 2026 NVIDIA 2026↓GR00T-WholeBodyControl (GEAR-SONIC whole-body controller workflow)NVIDIA, 2026; as of September 2026, Figure and DeepMind have published no code or weights for theirs.

One VLA, feet to fingertips: Gemini Robotics 2

Google DeepMind removed the boundary. Gemini Robotics 2 is a single frontier VLA controlling an Apptronik Apollo 2 from feet to fingertips under one policy, dynamically managing center of mass so the robot steps, squats, and bends instead of reaching from a static base, and driving a 22-DoF five-fingered SharpaWave hand as well as parallel grippers on a Franka Duo. The same checkpoint runs across Apollo 2 with SharpaWave hands, Apollo 2 with Inspire hands, and Franka Duo with Robotiq grippers, and the On-Device variant adapts to a new bi-arm embodiment in a few hours, typically with under 200 examples Parada 2026↓Gemini Robotics 2 brings whole body intelligence to robotsCarolina Parada, 2026. The trade-off is opacity: the action representation, control frequency, and parameter counts are all undisclosed in the announcement, so the decomposition can be verified only through its published behavior. The published numbers are where that verification starts.

The Gemini Robotics 2 numbers, as published

DeepMind published per-task success rates with the announcement, and they are the most honest snapshot of where whole-body capability actually is. Whole-body picking succeeds most of the time and falls with height: 76.3% from a shelf, 68.4% from a table, 45.7% from the floor, measured on Apollo 2 with Inspire hands. Gripper-based precise insertion reaches 89.6% on the Franka Duo. Multi-finger dexterity on the Apollo 2 Sharpa hand spans 92% (unscrewing a bulb) down to 32% (dustpan), a 60-point spread on the same hand under the same policy, and the vendor's own caption calls multi-finger manipulation the remaining challenge Parada 2026↓Gemini Robotics 2 brings whole body intelligence to robotsCarolina Parada, 2026.

What tracking does not solve

Three limits carry forward into everything built on this lineage. First, motion tracking inherits the human demonstrator's envelope. A tracking policy reproduces motion; it does not decide what task the motion should accomplish, and goal-directed loco-manipulation still has to come from somewhere upstream. Second, retargeting is lossy. Human kinematics are not robot kinematics, and morphologically infeasible references remain a persistent failure source, which is the explicit motivation of the 2026 robustness work Ma 2026↓Robust and Generalized Humanoid Motion TrackingYubiao Ma, Han Yu, Jiayin Xie et al., 2026. Third, the frontier results are vendor communications. Helix 02's 61-action dishwasher run and Gemini Robotics 2's success table both lack papers, baselines, and external replication.

There is also a structural question the three decompositions answer differently: whether balance and contact belong inside the foundation model or beneath it. Figure and NVIDIA both say beneath it, and disagree only on whether the interface is a tracked reference motion or a latent token. DeepMind says inside it, one policy for everything, and its own numbers show the cost: whole-body tasks succeed at 45.7 to 76.3% where specialized gripper insertion reaches 89.6%. Whether that gap closes with scale or is the price of the unified decomposition is the open empirical question.