Action Chunking (ACT and ALOHA)

Behavior cloning fails in closed loop for a precise reason: a policy with per-step error ε on the expert's state distribution can accumulate cost that grows quadratically with episode length, because each mistake carries the robot into states the expert never visited Ross 2011A Reduction of Imitation Learning and Structured Prediction to No-Regret Online LearningStéphane Ross, Geoffrey J. Gordon, J. Andrew Bagnell, AISTATS 2011, 2011. Over a T-step episode the worst-case cost scales as

where the second bound is what DAgger buys by repeatedly relabeling the states the policy actually visits Ross 2011A Reduction of Imitation Learning and Structured Prediction to No-Regret Online LearningStéphane Ross, Geoffrey J. Gordon, J. Andrew Bagnell, AISTATS 2011, 2011. Action chunking attacks the same bound from a different side: it shrinks the effective horizon. Instead of predicting one action per inference, the policy predicts a sequence of k actions at once, so the number of sequential decisions in an episode drops by a factor of k.

ACT: chunking with a CVAE

Action Chunking with Transformers (ACT), introduced together with the ALOHA bimanual teleoperation hardware, wraps chunking in a conditional variational autoencoder Zhao 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost HardwareTony Z. Zhao, Vikash Kumar, Sergey Levine et al., RSS 2023, 2023. During training, an encoder reads the demonstration's action sequence and produces a latent style variable. At test time the policy decodes a chunk conditioned on camera images, joint positions, and the latent clamped to zero.

The chunk-size dial

Chunk size is a real bias/variance dial, not a hyperparameter footnote. On the ALOHA insertion and transfer tasks, ACT measures roughly 1% success at k=1, 44% at k=100, and a decline at k=200 and k=400 as the policy becomes effectively open-loop Zhao 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost HardwareTony Z. Zhao, Vikash Kumar, Sergey Levine et al., RSS 2023, 2023. Too small and compounding error returns; too large and the policy stops reacting to the world mid-chunk.

Temporal ensembling and its limits

ACT executes chunks with temporal ensembling: at each timestep, the actions predicted for that step by overlapping chunks are averaged with exponential weights $w_i = \exp(-m \cdot i)$, where i counts how many chunks ago the prediction was issued. This smooths chunk boundaries at the cost of one inference per step instead of one per chunk.

# Temporal ensembling (ACT, arXiv:2304.13705). preds[i] is the prediction
# for the current action made by the chunk issued i steps ago.
def temporal_ensemble(preds: list[np.ndarray], m: float = 0.01) -> np.ndarray:
    weights = np.exp(-m * np.arange(len(preds)))   # w_i = exp(-m * i)
    weights /= weights.sum()
    return np.tensordot(weights, np.stack(preds), axes=1)

Real-Time Chunking (RTC) reframes the hand-off between chunks as an inpainting problem on the flow or diffusion trajectory: actions that will already have executed are frozen, the overlapping middle is partially attended to, and the remainder is generated fresh Black 2025Real-Time Execution of Action Chunking Flow PoliciesKevin Black, Manuel Y. Galliker, Sergey Levine, 2025. RTC holds throughput flat out to +200 ms of injected inference delay with no training-time change, which is why later flow-matching policies adopt it over ensembling.

The dial below injects inference delay into both execution strategies. Temporal ensembling holds while inference stays fast, then the averaged action drifts off the committed mode and the strategy fails inside the documented 100 to 200 ms window. RTC freezes the actions that will already have executed and stays on the in-flight mode, so its throughput line does not move Black 2025Real-Time Execution of Action Chunking Flow PoliciesKevin Black, Manuel Y. Galliker, Sergey Levine, 2025.

The lineage in numbers

Chunking started as a fix for compounding error and became standard infrastructure. The table sorts the major policies by how far ahead they predict and how fast they run; weights marked open can be downloaded, and the open releases stop at pi0.5. Cells marked n/a are undisclosed or embodiment-dependent, not zero.

Where the idea travels next

Mobile ALOHA extends the same recipe to a wheeled whole-body platform and shows that co-training on static ALOHA data lifts mobile manipulation success substantially Fu 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body TeleoperationZipeng Fu, Tony Z. Zhao, Chelsea Finn, 2024. Later modules in this domain cover how diffusion policies and flow-matching models generate the chunks themselves, and how vision-language-action models schedule them at high control frequency.