Manipulation & Learned Policies
Real-Time Execution
Temporal ensembling, real-time chunking, and the latency budgets that decide whether the control loop closes.
Last reviewed 2026-08-08
A policy that needs 60 ms to think cannot run a 50 Hz robot. The control period at 50 Hz is 20 ms, and every component in the loop, cameras, network, preprocessing, model, spends from that budget. The earlier modules treated action chunking as an algorithmic choice; this module is about what happens when the algorithm meets the clock. Real-time execution turns out to be a first-class architectural problem, not an engineering afterthought: the way a policy hands off from one chunk to the next while inference is still running changes what the robot physically does, and the naive ways to do it fail in documented, measurable ways Black 2025.
Where the milliseconds go
The Real-Time Chunking paper publishes the most detailed latency breakdown available for a deployed VLA system. On a mobile manipulator, the measured total was 139 ms per inference: 97 ms of model compute, 21 ms of network transit, 11 ms of image resizing, and 9.7 ms of everything else. On a static robot the total was 108 ms Black 2025. Two things stand out. The model dominates, but not overwhelmingly: a fifth of the budget is spent moving and preparing data. And both totals are five to seven control periods long. A 50 Hz robot controlled this way is not closing a loop at 50 Hz; it is executing open-loop chunks and re-planning at 5 to 9 Hz.
NVIDIA's VLA-Perf study, the first systematic latency analysis of VLAs across edge and cloud hardware, adds the scaling picture NVIDIA Research 2026. Its central finding is that denoising steps dominate inference time for diffusion and flow-based policies, while chunk size contributes negligibly. That is why the whole field spends its effort on step count and hand-off strategy rather than on shorter chunks.
The hardware wall
How big a model can close the loop at all? On an H100 server, pi0 runs end-to-end at 162.5 Hz, comfortably inside any budget. On a Jetson Thor, the same model manages 19.0 Hz (52.57 ms), and the 9.1B-parameter pi0-L drops to 3.9 Hz NVIDIA Research 2026. The interactive below puts a model-size slider on that curve: drag it and watch the inference bar cross the 20 ms budget line. The reference list underneath pins the other sourced numbers from this module against the same budget.
does not close at 50 Hz: 19 Hz effective, 2 deadlines missed
Anchors are the VLA-Perf measurements on Jetson Thor (arXiv:2602.18397): pi0 at 52.6 ms (19 Hz) and pi0-L at 3.9 Hz. The scaling between and below them is an illustrative memory-bound model, not a measurement.
- pi0, H100 server6.2 ms162.5 Hz end-to-end, VLA-Perfcloses at 50 Hz
- pi0, Jetson Thor52.6 ms19.0 Hz end-to-end, VLA-Perfover budget
- pi0.6, H100 server63 ms5 denoising steps, 3 cameras, per chunkover budget
- RTC total, static robot108 msmodel + network + preprocessing, measuredover budget
- RTC total, mobile robot139 ms97 model + 21 network + 11 resize + 9.7 otherover budget
- pi0.7 tolerated latency240 mstraining-time RTC, 12 ticks at 50 Hztolerated by design
VLA-Perf's takeaways are sobering for on-robot deployment NVIDIA Research 2026. Roughly 10 Hz control is achievable on Jetson Thor for pi0-scale models, but 100 Hz control requires model-level architectural changes, not just faster hardware. Everything on the Thor is memory-bound, so latency tracks parameter count. Quantization to FP4 or FP8 buys a 2 to 4x latency reduction with minimal accuracy loss. And server-side inference beats on-device inference in almost all scenarios except very poor networks, which is why the latency budget includes a network line at all.
Three ways to hand off a chunk
Suppose inference takes d milliseconds. The policy has emitted a chunk, the robot is executing it, and a new chunk is on the way. What happens at the boundary is a design decision, and there are three options on the table.
Synchronous execution waits: run the current chunk to completion, pause while the next one is computed, then start it. This is how the original pi0 release executed its 50-step chunks Black 2024. The motion is smooth, but the pause itself is a problem: the robot decelerating to a stop and holding is a state the training demonstrations never contained, so each hand-off throws the policy off-distribution. The RTC paper reports that removing the pauses improved precision along with throughput, because the pauses change the robot's dynamics in a way the model does not model Black 2025.
Naive switching never waits: the instant the new chunk arrives, execution jumps to it. There is no dead time, but there is a discontinuity. The new chunk was inferred from an observation d milliseconds old, the world has moved since, and the two chunks disagree about what to do right now. The command steps by the full disagreement in a single control tick: a jerk event the hardware feels.
Temporal ensembling, ACT's answer from 2023, queries the policy at every timestep and averages the overlapping predictions with exponential weights, newer observations dominating Zhao 2023. At low latency this is smooth and reactive. At real VLA latencies it fails in a specific way: when two overlapping chunks commit to different modes (left of the obstacle versus right), the weighted average lands between the modes, in a place where no valid action exists. Physical Intelligence documents catastrophic failures at +100 ms and +200 ms of injected delay Physical Intelligence 2025. The action chunking module has an interactive for that failure mode.
Real-Time Chunking reframes the hand-off as inpainting, and it is the only option that pays neither cost Black 2025. If inference takes d controller ticks, the first d actions of the new chunk can never be executed, because that time will already have passed; freeze them to the values from the old chunk that will actually have run. For the remaining overlap, apply partial attention, softly biasing the new chunk toward the old one, more strongly for earlier timesteps. The rest generates freely. Diffusion and flow models are natively good at inpainting, so RTC drops onto pi0 and pi0.5 with no training-time change. Measured throughput stays flat from +0 ms to +200 ms of injected delay, and RTC completed precise dynamic tasks, striking a match, plugging in an Ethernet cable, at over 300 ms of delay Black 2025.
The comparison below makes the three behaviors numeric. One delay slider drives all three traces, and each panel reports its peak per-tick velocity step against a stated jerk limit: synchronous stays smooth but pays the delay as dead time, the naive switch spikes past the limit once the delay reaches 100 ms, and RTC holds the limit across the whole range.
d = 0 ms: all three modes track smoothly; jerk limit 0.30 per 20 ms tick
The traces model the published behaviors (arXiv:2506.07339): synchronous execution pays dead time, naive switching pays a discontinuity that grows with delay, and real-time chunking pays neither. They are a model, not measured robot data; the 0.30 jerk limit is an illustrative threshold, stated so the comparison is numeric.
Training the tolerance in
RTC's inference-time trick has a training-time successor. Instead of only freezing and blending at deployment, the model is trained with the in-flight actions as conditioning input, so it learns to continue its own committed trajectory rather than merely being constrained to Physical Intelligence 2025. pi0.7 uses this: during training it simulates delays of 0 to 12 controller ticks, which buys a maximum tolerated inference latency of 240 ms on a 50 Hz robot, twelve times the control period Physical Intelligence 2026. That margin is what makes a 5B-parameter policy with 63 ms of chunk inference viable in production: the loop does not have to close at 50 Hz for the behavior to stay continuous, because the policy was trained to expect its own latency.
Spending the step budget
Since denoising steps dominate latency NVIDIA Research 2026, the step count is where the frontier labs spend down. pi0 integrated its flow with 10 Euler steps per chunk Black 2024. pi0.6 cut that to 5, landing at 63 ms per chunk on a single H100 with three cameras Physical Intelligence 2025. pi0.7 stays at 5 Physical Intelligence 2026. The research line that collapses generation to one or two steps, consistency distillation and its relatives Prasad 2024, has not been adopted by the frontier labs for their flagship policies: they reduced steps from 10 to 5 and attacked the hand-off problem with RTC instead. One-step distillation still matters, but mostly for the opposite end of the stack, on-robot edge deployment where there is no H100 and the 20 ms budget is a hard wall rather than a tolerated overrun.
How to read these numbers
The latency figures above come from two arXiv papers and one vendor model card; the pi0.6 figure is vendor-reported. Both interactives are models anchored to those published numbers, not re-runs of the experiments: the control-loop scaling between the two VLA-Perf anchors is illustrative, and the hand-off traces encode the published qualitative behaviors with a stated, illustrative jerk threshold. The sourced numbers themselves (52.57 ms, 63 ms, 108 and 139 ms, 240 ms) are measured or specified by their respective papers.
The pi line module traces how the step count and the RTC machinery evolved across the pi0 family, and the comparison matrix puts the resulting inference costs next to the other method families.