Hierarchical Approaches

Every long-horizon robot system has to answer the same organizational question: what decides what to do next, and what decides how to move? The modules so far covered the how (chunks, diffusion, flow matching, action tokens). This module is about the what, and about a structural shift that is easy to miss if you only read 2026 model cards. In 2022 the what lived outside the policy, in a separate planner that dispatched to a library of skills. In 2026 the what lives inside the network, as a low-frequency inference pass in the same weights that emit actions. The separate-planner architecture is the lineage that lost.

The interactive below draws four 2025 to 2026 systems on a schematic two-second timeline. It combines source-reported output rates with explicitly labelled drawing assumptions. Dragging the playhead counts events in that illustration; it does not replay a measured robot schedule. A task-instruction pulse is an initial condition, not a reported arrival rate.

SayCan: scoring usefulness and affordance

SayCan (2022) scores candidate skills from a supplied library. The LLM scores each skill description as a next step for the instruction; an affordanceaffordanceAn estimate of whether an available skill can succeed in the current situation. SayCan combines this estimate with a language model's score for the skill's usefulness to the instruction. Its learned picking value functions require empirical calibration, and its navigation and placement affordances also use hand-designed rules; a high score is not confirmation of successful execution. score estimates whether that skill can succeed in the current state. SayCan multiplies the scores, executes the highest-scoring skill's policy, appends the selected skill description to the prompt, and repeats until a termination option is selected Ahn 2022↓Do As I Can, Not As I Say: Grounding Language in Robotic AffordancesMichael Ahn, Anthony Brohan, Noah Brown et al., 2022.

In the mobile-manipulator implementation, picking uses a multitask, language-conditioned behavioral-cloning policy and RL-trained value functions, not a separately trained model for every skill. The value functions require empirical calibration before use as probabilities. Navigation and placing also use hand-designed policies or affordance rules Ahn 2022↓Do As I Can, Not As I Say: Grounding Language in Robotic AffordancesMichael Ahn, Anthony Brohan, Noah Brown et al., 2022.

The available skills limit what the system can do. Environmental feedback reaches this version's planner only through the value functions at the current decision step, and the paper reports difficulty reacting when a skill fails despite a high value. Appending a selected skill is therefore not confirmation that it succeeded Ahn 2022↓Do As I Can, Not As I Say: Grounding Language in Robotic AffordancesMichael Ahn, Anthony Brohan, Noah Brown et al., 2022.

Code as Policies: the planner writes programs

Code as Policies (2022) prompts a code-writing LLM with example instruction-program pairs to generate Python that calls supplied perception and control APIs. A function-generation prompt recursively expands functions missing from the execution scope. The programs can combine loops, conditionals and library calls for spatial arithmetic, and can express feedback loops over perceptual outputs Liang 2022↓Code as Policies: Language Model Programs for Embodied ControlJacky Liang, Wenlong Huang, Fei Xia et al., ICRA 2023, 2022.

These capabilities depend on the supplied APIs and prompts. The paper assumes the instructions are feasible and cannot establish in advance whether a generated response is correct. Its real-robot systems use ViLD or MDETR VLMvision-language modelA model pretrained jointly on web-scale image and text data, so visual recognition and language semantics live in one set of weights. RT-2's bet was that this pretraining is an asset for robot control: co-fine-tuning a vision-language model on robot trajectories and its original web data together transfers semantic knowledge, recognizing objects and following instructions the robot data never covered, into the policy. The vision-language-action models that followed all start from such a backbone. detections, with depth and a camera-to-robot transform registered in advance to lift detections into 3D. The authors attribute many real-world failures to inaccurate detections. Quantitative robot-system evaluation is confined to the specified tabletop simulation; the real-robot systems are demonstrations Liang 2022↓Code as Policies: Language Model Programs for Embodied ControlJacky Liang, Wenlong Huang, Fei Xia et al., ICRA 2023, 2022.

The keypoint turn: VLMs emit geometry, not skill names

These methods put different geometric interfaces between a vision-language model and robot motion:

• MOKA uses GroundedSAM masks to mark candidate points in the observed image and prompts a pretrained VLM to select grasp, function and target keypoints where applicable, plus waypoint regions and motion attributes. The perception pipeline supplies calibrated depth for lifting surface points into the robot frame; a separate grasp sampler and motion pipeline execute the selections. This does not require a full 6-DoF object-pose estimate as an intermediate stage. The paper also studies in-context examples and distillation into a robot policy Liu 2024↓MOKA: Open-World Robotic Manipulation through Mark-Based Visual PromptingFangchen Liu, Kuan Fang, Pieter Abbeel et al., 2024.
• ReKep proposes semantic keypoints using DINOv2 features within SAM masks and calibrated RGB-D geometry. GPT-4o writes stage-specific Python sub-goal and path constraints over those keypoints; a separate hierarchical solver replans end-effector motions and penalizes constraint violations. This is not a guarantee of physical feasibility: tracking and rigidity assumptions matter, and some evaluations use human annotations or disable tracking Huang 2024↓ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic ManipulationWenlong Huang, Chen Wang, Yunzhu Li et al., 2024.
• RoboPoint instruction-tunes a VLM to predict image-space spatial-affordance points, co-training synthetic object and free-space reference data with real-image VQA and LVIS detection data. On Where2Place, a benchmark of 100 real-world images, Table 2 reports 46.77% point-in-mask accuracy for RoboPoint and 29.06% for GPT-4o, as means over three runs. This measures predicted points inside the annotated target mask, not robot grasp success; the robot experiments separately use depth, an end-effector offset and a motion planner Yuan 2024↓RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for RoboticsWentao Yuan, Jiafei Duan, Valts Blukis et al., 2024.

ECoT trains a VLAvision-language-action modelA robot policy built by adapting a vision-language model, pretrained on web-scale image and text data, to output robot actions. RT-2 established the approach by representing actions as text tokens and co-fine-tuning on web and robot data together, showing that semantic knowledge from web pretraining transfers into control. The same recipe underlies OpenVLA and the π series of generalist policies. to generate task plans, subtask reasoning, movement descriptions, object bounding boxes and the current gripper's pixel position before action tokens. It adds synthetic reasoning annotations to existing Bridge V2 demonstrations, not new physical robot trajectories Zawalski 2024↓Robotic Control via Embodied Chain-of-Thought ReasoningMichał Zawalski, William Chen, Karl Pertsch et al., CoRL 2024.

The paper reports a 28-percentage-point absolute success-rate improvement over OpenVLA trained on the same Bridge V2 data. In Table 1, ECoT reaches 66% versus 44% in the in-distribution camera view and 64% versus 30% in the out-of-distribution view. The suite includes familiar tasks, spatial relations, unseen objects and unseen instructions; the paper reports 314 trials per approach and per-view means with one standard error, not a general benchmark guarantee Zawalski 2024↓Robotic Control via Embodied Chain-of-Thought ReasoningMichał Zawalski, William Chen, Karl Pertsch et al., CoRL 2024.

The main evaluation already keeps high-level reasoning fixed for five steps. A separate variant in the inspected v3 paper predicts four future gripper positions as a motion trace; that is not the recipe behind the main Table 1 comparison. The authors describe these predictions as an imperfect proxy for behavior, and a generated plan can differ from the action ultimately selected. Reasoning-token generation remains a speed limitation Zawalski 2024↓Robotic Control via Embodied Chain-of-Thought ReasoningMichał Zawalski, William Chen, Karl Pertsch et al., CoRL 2024.

2026: hierarchy moved inside the network

Read the current frontier systems side by side and the same pattern appears in each one.

• π0.5 co-trains on robot data, semantic subtask labels, and multimodal web tasks. Pretraining uses discrete tokens; post-training adds the continuous action expert. At inference, the same model predicts a high-level language subtask and then conditions low-level actions on it. High-level inference runs less frequently than low-level action inference; this does not establish a numerical inference rate Black 2025↓π0.5: a Vision-Language-Action Model with Open-World GeneralizationPhysical Intelligence, Kevin Black, Noah Brown et al., 2025.
• Hi Robot uses two separately trained policies initialized from PaliGemma: a high-level VLM and a low-level π0 VLA. The high level reads camera images and open-ended instructions, then sends simpler language commands to the low level, which also receives images and robot state. High-level inference is rerun when one second has elapsed or new human feedback arrives; users can intervene with text or transcribed speech and can signal a return to the prior task Shi 2025↓Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action ModelsLucy Xiaoyang Shi, Brian Ichter, Michael Equi et al., ICML 2025. The training combines human-labelled demonstrations with synthetic interactions, with a separate high-level policy for each task domain in the reported experiments. Evaluations cover table bussing, sandwich making and grocery shopping. This hierarchy does not guarantee successful execution: the authors report missing high-level memory, low-level instruction violations and difficulty recovering from dropped objects Shi 2025↓Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action ModelsLucy Xiaoyang Shi, Brian Ichter, Michael Equi et al., ICML 2025.
• Gemini Robotics 1.5 combines both patterns: with thinking enabled, the VLA appends natural-language reasoning to its context before emitting actions; the agentic system pairs it with a separate GR-ER 1.5 orchestrator. ER's thinking-token budget is varied in the report's embodied-reasoning evaluations, with task-dependent benefits. Internal VLA thinking therefore does not mean that the external orchestrator has disappeared Gemini Robotics Team 2025↓Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion TransferGemini Robotics Team, Abbas Abdolmaleki, Saminda Abeyruwan et al., 2025.
• GO-2 is described by AgiBot as a unified system with an asynchronous two-module hierarchy. Its Semantic Planning Module (System 2) generates action intents through Progressive Refinement at a lower frequency; its Action Following Module (System 1) combines intents and real-time observations through Residual Refinement at a higher frequency. The announcement does not give numerical rates or establish that the modules share one set of weights. Teacher forcing is a training mechanism, not a third runtime layer AgiBot 2026↓The Unity of Reasoning and Action: AGIBOT Unveils Genie Operator-2 (GO-2) Next-Gen Embodied Foundation ModelAgiBot, 2026.
• Helix 02 extends Figure’s S2/S1 architecture with a learned S0 controller. Figure describes S2 semantic behavior sequencing, S1 full-body joint targets at 200 Hz from vision, touch, and proprioception, and a 10M-parameter S0 network that outputs joint-level actuator commands at 1 kHz. These are vendor-reported rates for different stages, not comparable inference-throughput results; the announcement does not specify S2’s numerical rate Figure AI 2026↓Introducing Helix 02: Full-Body AutonomyFigure AI, 2026.
• π0.7 can condition its actions on visual subgoal images. At runtime a BAGEL-initialized world model produces these images from observations and the subtask instruction. Subtask and image generation run in separate threads, while VLA inference uses the latest available outputs Ai 2026↓π0.7: a Steerable Generalist Robotic Foundation Model with Emergent CapabilitiesBo Ai, Ali Amin, Raichelle Aniceto et al., 2026. π0.6-MEM uses a high-level policy to predict the next language subtask and an updated language-memory summary from previous memory, observations, and the task. Its low-level policy conditions actions on the subtask and a shorter observation history. This is MEM’s memory mechanism, not a claim that it shares π0.7’s runtime implementation or a disclosed update frequency Torne 2026↓MEM: Multi-Scale Embodied Memory for Vision Language Action ModelsMarcel Torne, Karl Pertsch, Homer Walke et al., 2026.

The π0.5 paper reports 50 Hz target commands for its mobile-manipulator setup and predicts 50 actions per chunk, but that does not establish one chunk inference per second. The interactive keeps 1 chunk/s only as a schematic cadence. Figure reports Helix 02 S1 joint targets at 200 Hz and S0 actuator commands at 1 kHz. GO-2’s lower/higher module frequencies and the other schematic intervals are not numerical rate measurements. The extra GO-2 motor-tick lane is a drawing device, not a third disclosed module Black 2025↓π0.5: a Vision-Language-Action Model with Open-World GeneralizationPhysical Intelligence, Kevin Black, Noah Brown et al., 2025 Figure AI 2026↓Introducing Helix 02: Full-Body AutonomyFigure AI, 2026 AgiBot 2026↓The Unity of Reasoning and Action: AGIBOT Unveils Genie Operator-2 (GO-2) Next-Gen Embodied Foundation ModelAgiBot, 2026.

The verdict

Two claims are worth stating plainly, and both need framing.

First: the five cited examples place high-level guidance at different interfaces. π0.5 uses the same model to infer a language subtask and actions conditioned on it Black 2025↓π0.5: a Vision-Language-Action Model with Open-World GeneralizationPhysical Intelligence, Kevin Black, Noah Brown et al., 2025. The π0.6-MEM report describes a high-level policy that predicts the next subtask and updated language memory, with a low-level policy conditioned on that subtask and recent observations Torne 2026↓MEM: Multi-Scale Embodied Memory for Vision Language Action ModelsMarcel Torne, Karl Pertsch, Homer Walke et al., 2026. π0.7 can use subgoal images from a BAGEL-initialized world model; when those images are used, subtask and image generation run asynchronously in separate threads Ai 2026↓π0.7: a Steerable Generalist Robotic Foundation Model with Emergent CapabilitiesBo Ai, Ali Amin, Raichelle Aniceto et al., 2026. With thinking enabled, Gemini Robotics 1.5 appends natural-language reasoning to the VLA context before actions, while its agentic system retains a separate GR-ER 1.5 orchestrator Gemini Robotics Team 2025↓Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion TransferGemini Robotics Team, Abbas Abdolmaleki, Saminda Abeyruwan et al., 2025. AgiBot describes GO-2 as a lower-frequency semantic planning module paired with a higher-frequency action-following module AgiBot 2026↓The Unity of Reasoning and Action: AGIBOT Unveils Genie Operator-2 (GO-2) Next-Gen Embodied Foundation ModelAgiBot, 2026. This is a comparison of reported mechanisms, not evidence that separate planners have been superseded across the field.

Second: geometric supervision in training does not establish the disappearance of geometric reasoning at runtime. π0.5’s v1 paper describes training the model to predict relevant bounding boxes before subtask labels, and adding indoor-scene and household-object data with bounding-box annotations to its web-data mixture Black 2025↓π0.5: a Vision-Language-Action Model with Open-World GeneralizationPhysical Intelligence, Kevin Black, Noah Brown et al., 2025. Gemini Robotics-ER 1.5 also reports complex pointing, including point sequences that can represent motion trajectories and paths Gemini Robotics Team 2025↓Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion TransferGemini Robotics Team, Abbas Abdolmaleki, Saminda Abeyruwan et al., 2025. That is evidence of a geometric reasoning capability, not proof that a specific deployed controller uses a keypoint loop. The scope here is those reported mechanisms, not an exhaustive inventory of 2026 frontier systems; no field-wide absence of runtime keypoint interfaces is claimed.