Manipulation & Learned Policies
Hierarchical Approaches
SayCan, code-as-policies, and keypoint affordances; why separate planners gave way to internalized hierarchy.
Last reviewed 2026-08-08
Every long-horizon robot system has to answer the same organizational question: what decides what to do next, and what decides how to move? The modules so far covered the how (chunks, diffusion, flow matching, action tokens). This module is about the what, and about a structural shift that is easy to miss if you only read 2026 model cards. In 2022 the what lived outside the policy, in a separate planner that dispatched to a library of skills. In 2026 the what lives inside the network, as a low-frequency inference pass in the same weights that emit actions. The separate-planner architecture is the lineage that lost.
SayCan: grounding a language model in what is physically possible
SayCan (2022) is the canonical separate-planner system. An LLM scores how useful each skill in a fixed library is for the instruction; a learned affordance value function, trained per skill, scores how possible that skill is right now, given the scene and the robot's state. The two scores multiply, the argmax executes, and the loop repeats with the completed skill appended to the prompt Ahn 2022.
The affordance half is the contribution that survived. An LLM asked to "clean up the spill" will happily plan around a sponge that is not in the room; the value function is what ties the plan to the current scene. The limitation is the other half: the planner can only choose among skills someone already trained, and it re-plans one skill at a time, so recovery from a failed skill means another expensive pass through the loop.
Code as Policies: the planner writes programs
Code as Policies (2022) replaced skill selection with code generation. The LLM writes executable Python that calls perception and control APIs, recursively defining any function it references but has not been given. That buys loops, conditionals, arithmetic over spatial quantities, and composition of primitives, an expressiveness a flat skill selector cannot reach Liang 2022. The failure modes moved too: the generated program is only as good as the APIs it calls, and a VLM's spatial estimates were the weakest link in the chain.
The keypoint turn: VLMs emit geometry, not skill names
The 2024 line of work kept the separate-planner structure but changed the interface. Instead of picking a skill or writing control code, the model emits a geometric quantity that a classical controller consumes. Three papers define the space:
- MOKA annotates the observed image with candidate marks and asks the VLM to select grasp keypoints, function keypoints, and target keypoints, then converts the selected points into a motion. The trick is that free-form manipulation becomes a visual question-answering problem, which the VLM is already good at Liu 2024.
- ReKep has the VLM write Python functions over sets of semantic keypoints that evaluate to a numerical cost; a solver then optimizes robot actions subject to those constraints, across space and time. This is the most classical-robotics design of the three: the VLM specifies the optimization problem, not the motion Huang 2024.
- RoboPoint fine-tunes a VLM specifically to predict spatial affordance points, trained on synthetic point-annotation data rather than real robot rollouts, and reports a 21.8% gain in affordance-prediction accuracy over GPT-4o with visual prompting Yuan 2024.
The bridge between this line and what came next is ECoT (2024), which moved the reasoning into the VLA: the model interleaves subtask decomposition, bounding boxes for task-relevant objects, and 2D motion traces with its action prediction, improving OpenVLA's generalization success by 28% absolute with no extra robot data Zawalski 2024. Its weakness was speed, since autoregressive reasoning tokens are generated serially before every action. Making that internal reasoning cheap is what the 2025-2026 systems figured out.
2026: hierarchy moved inside the network
Read the current frontier systems side by side and the same pattern appears in each one.
- π0.5 trains one network on hybrid examples that interleave images, language commands, object detections, semantic subtask prediction, and actions. At inference the model first predicts a high-level language subtask at low frequency, then conditions the action expert on that subtask at high frequency. There is no separate planner model Black 2025.
- Hi Robot is the explicit two-model version from the same lab: a high-level VLM emits language subtasks to a low-level VLA and handles open-ended instructions and mid-task human feedback. It reads like the transitional form between SayCan and π0.5: the hierarchy still spans two networks, but both are learned VLMs rather than an LLM sitting above a hand-built skill library Shi 2025.
- Gemini Robotics 1.5 runs both patterns at once: the VLA interleaves language thinking traces with its actions, while ER 1.5 remains a separate high-level orchestrator with a tunable thinking budget Gemini Robotics Team 2025.
- GO-2 splits the hierarchy by frequency instead of by model: an action chain-of-thought planner emits a macro plan of intents at low frequency, and an asynchronous high-frequency follower refines them against live observations AgiBot 2026.
- Helix 02 goes furthest down the stack: three learned layers, S2 sequencing behaviors, S1 mapping all sensors to all joints at a vendor-reported 200 Hz, and S0, a 10M-parameter whole-body controller at a vendor-reported 1 kHz Figure AI 2026.
- π0.7 adds a world model generating visual subgoal images as the intermediate representation between language and action, which is arguably the successor to the keypoint line, with generated images replacing hand-annotated geometry Physical Intelligence 2026. MEM extends the same low-frequency step to also emit a memory update, so the high level manages state as well as intent Torne 2026.
The interactive below draws four of these systems as lanes on one two-second timeline. Scrub the playhead and watch which lanes fire: the motor lanes tick constantly, the subtask lanes barely move. Switching systems shows the same pattern instantiated four ways, including Helix 02's 200 Hz and 1 kHz lanes and GO-2's asynchronous planner/follower split. Rates tagged "schematic" are not stated in the primary source; the lane exists because the architectural split is disclosed, but the exact frequency is not.
amber ticks: updates fired at or before the playhead. dim ticks: pending. Rates tagged (schematic) are not stated in the primary source.
- Task instructiononce1 update, last update: 0 ms
Given once at episode start, e.g. "clean the kitchen".
- Subtask prediction~1 Hzschematicwaiting for first update
High-level inference inside the same network emits the next language subtask. The paper states low frequency; the exact rate is not disclosed, 1 Hz shown schematically.
- Chunk inference1 chunk/swaiting for first update
50-step action chunks at 50 Hz cover one second each; the original release runs inference synchronously.
- Motor commands50 Hzwaiting for first update
One action per control tick, executed from the current chunk.
One network, hierarchy internalized: the same VLA predicts the next language subtask at low frequency and conditions the flow-matching action expert on it at high frequency. There is no separate planner model.
The verdict
Two claims are worth stating plainly, and both need framing.
First: separate-planner architectures of the SayCan type have been superseded by internalized hierarchy in the 2026 frontier systems. No single source states this. It is a synthesis, the consistent reading across π0.5, π0.6, π0.7, Gemini Robotics 1.5, and GO-2, every one of which generates its subtasks, thinking traces, or intents inside (or asynchronously alongside) the same learned stack that produces actions Black 2025 Gemini Robotics Team 2025 AgiBot 2026. The reasons are structural rather than ideological: a separate planner cannot share representations with the policy, re-planning through a dispatch loop is slow, and a fixed skill library caps what the system can express. Gemini Robotics is the partial exception: Google keeps ER as a separate orchestrator for multi-minute, multi-robot coordination while the per-robot hierarchy stays internal Gemini Robotics Team 2025.
Second: the keypoint and affordance methods did not die; they moved from runtime pipelines into training data. π0.5 co-trains on bounding-box prediction and keypoint prediction as auxiliary objectives, which is the MOKA and RoboPoint idea absorbed into the mixture rather than running as a separate stage at inference Black 2025. The 2024 papers remain the clearest statement of why geometric intermediate representations help, and their runtime incarnations are still reasonable for a research prototype with a frozen VLM. But no 2026 frontier system routes its control loop through a runtime keypoint interface.
How to read this synthesis
The supersession claim in this section is the atlas's own reading across five systems, not a quote from any one of them. The per-system facts underneath it (subtask prediction in π0.5, interleaved thinking in Gemini Robotics 1.5, the asynchronous split in GO-2, the S0/S1/S2 rates in Helix 02) are each cited to their primary source. If a 2027 system ships a competitive separate planner, this section is where the retraction goes.
The comparison matrix encodes this shift as the hierarchy axis across all of the methods in the atlas, and the pi line traces how one lab moved from Hi Robot's two networks to π0.5's single pass in about a year.