Vision-Language-Action Models

The modules so far covered policies trained from scratch on robot data alone: behavior cloning, ACT, and Diffusion Policy. The vision-language-action (VLA)vision-language-action modelA robot policy built by adapting a vision-language model, pretrained on web-scale image and text data, to output robot actions. RT-2 established the approach by representing actions as text tokens and co-fine-tuning on web and robot data together, showing that semantic knowledge from web pretraining transfers into control. The same recipe underlies OpenVLA and the π series of generalist policies. line changes the starting point. Instead of training a policy from nothing, take a model that already understands images and language from web-scale pretraining, and teach it to emit robot actions. The design question that defines this lineage is how to get actions out of a model whose output space is text.

The interactive below is an illustrative seven-coordinate action sequence, not a recorded rollout or a literal tokenizer vocabulary. It places values in 256 uniform bins on a fixed normalized interval and uses synthetic token labels. Scrub the control step to inspect the assigned bin and reconstruction error. Quantization discards within-bin detail, but error can be zero when a value equals its reconstructed bin center. OpenVLA’s actual bin bounds use training-data quantiles; the toy does not estimate those statistics. Kim 2024↓OpenVLA: An Open-Source Vision-Language-Action ModelMoo Jin Kim, Karl Pertsch, Siddharth Karamcheti et al., 2024

RT-1: actions become tokens

RT-1 (2022) combines an ImageNet-pretrained EfficientNet with a pretrained instruction embedding and FiLM language conditioning. It tokenizes six images: TokenLearner reduces 81 visual tokens per image to eight, yielding 48 tokens for an eight-layer decoder-only transformer with 19M parameters. The paper reports 35M parameters for the full system. Brohan 2022↓RT-1: Robotics Transformer for Real-World Control at ScaleAnthony Brohan, Noah Brown, Justice Carbajal et al., 2022

Its 11 action coordinates comprise seven arm variables (translation, rotation and gripper opening), three base variables (x, y and yaw), and a switch between arm control, base control and episode termination. Each variable is discretized into 256 bins uniformly spaced within its bounds; training uses categorical cross-entropy over action tokens. Brohan 2022↓RT-1: Robotics Transformer for Real-World Control at ScaleAnthony Brohan, Noah Brown, Justice Carbajal et al., 2022

The paper reports roughly 130k training episodes covering more than 700 tasks, collected over 17 months with 13 robots, and 3,000 real-world evaluation trials. Its authors argue that generalization requires both scale and breadth, with sufficiently connected tasks to learn reusable patterns. That is their dataset argument, not a guarantee that pooling arbitrary demonstrations improves every task. Brohan 2022↓RT-1: Robotics Transformer for Real-World Control at ScaleAnthony Brohan, Noah Brown, Justice Carbajal et al., 2022

RT-1 reports a 3 Hz robot-control rate and predicts the next action rather than an action chunk. Its implementation uses a fixed wait after capturing the state to keep action timing consistent; Appendix C.1 gives 280 ms as the maximum observed latency of all components. These are settings of the reported system, not a universal limit on tokenized policies. Its visual and language encoders are pretrained, so describing the whole system as trained on robot data alone would also be misleading. Brohan 2022↓RT-1: Robotics Transformer for Real-World Control at ScaleAnthony Brohan, Noah Brown, Justice Carbajal et al., 2022

RT-2: actions become text

RT-2 (2023) attacked the knowledge problem directly Brohan 2023↓RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic ControlAnthony Brohan, Noah Brown, Justice Carbajal et al., 2023. Rather than bolting a robot head onto a vision-language modelvision-language modelA model pretrained jointly on web-scale image and text data, so visual recognition and language semantics live in one set of weights. RT-2's bet was that this pretraining is an asset for robot control: co-fine-tuning a vision-language model on robot trajectories and its original web data together transfers semantic knowledge, recognizing objects and following instructions the robot data never covered, into the policy. The vision-language-action models that followed all start from such a backbone., it expresses actions as strings in the VLM's existing vocabulary: each action dimension is discretized into 256 bins, and the bin indices are serialized into the token stream. The two backbones handle the vocabulary differently, which is a detail worth keeping: PaLI-X already has a token for every integer up to 1000, so bins map onto existing number tokens, while PaLM-E overwrites the 256 least frequently used tokens with the action vocabulary. The backbone (PaLI-X or PaLM-E, up to 55B parameters) is then co-fine-tuned on a mixture of robot trajectories and the original web vision-language data. The authors report that mixing the original web data with robot data during co-fine-tuning produces more generalizable policies than fine-tuning on robot data alone Brohan 2023↓RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic ControlAnthony Brohan, Noah Brown, Justice Carbajal et al., 2023.

RT-2 separates emergent object-choice behavior, such as selecting the smallest object or the one closest to another, from its additional chain-of-thought experiment. For the latter, the authors fine-tune a variant with an intermediate natural-language plan before the action tokens; their concrete example is “Instruction: I’m hungry. Plan: pick rxbar chocolate.” The abstract also gives choosing a rock as an improvised hammer as a multi-stage semantic-reasoning example. This is more specific than claiming that ordinary action decoding automatically supplies a reasoning trace. Brohan 2023↓RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic ControlAnthony Brohan, Noah Brown, Justice Carbajal et al., 2023

In their May 2025 research note, Danny Driess and colleagues at Physical Intelligence argue that this tokenized representation “is not suitable for high-frequency, precise, or fluent motions,” citing their π0 and π0-FAST experiments. Their analogy is “controlling your arm by verbally saying which muscles should contract.” This is the authors’ stated assessment of the representation, not an independently established impossibility result. Driess 2025↓VLAs that Train Fast, Run Fast, and Generalize BetterDanny Driess, Jost Tobias Springenberg, Brian Ichter et al., 2025

Open X-Embodiment and RT-X: pooling the data

Open X-EmbodimentOpen X-EmbodimentA 2023 collaboration across 34 labs that pooled 60 existing robot datasets into one standardized format: over a million trajectories spanning 22 robot embodiments and 527 skills. The project also trained the RT-X models on the pool, and found that a policy trained across many robots outperforms versions trained on each robot's data alone, with skills carrying over to embodiments the policy saw little of. pooled more than one million real robot trajectories spanning 22 embodiments, 527 skills, and 160,266 tasks Open X-Embodiment Collaboration 2023↓Open X-Embodiment: Robotic Learning Datasets and RT-X ModelsOpen X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman et al., 2023. The report’s abstract names 21 contributing institutions; Section III-A separately describes 60 constituent datasets from 34 research labs. These differently labelled populations should not be treated as interchangeable counts. The RT-1-X and RT-2-X experiments used a nine-embodiment training subset, not all 22 embodiments. Transfer depended on the model and evaluation: RT-1-X improved over the original methods on four of five small-data domains but underperformed the domain-specific RT-1 baseline in the large-data setting. RT-2-X acquired additional Google Robot skills from WidowX Bridge data, while performing roughly on par with RT-2 on the tested unseen-object, background, and environment generalization. These results support cross-embodimentcross-embodimentLearning from, or transferring to, robots with different morphologies: different joint counts, action dimensions, and sensor suites. The Open X-Embodiment project pooled data from 22 robot embodiments into a single training mixture and showed that the resulting RT-X policies carry skills across platforms. The gain is not free, because each body defines its actions differently, and the field answers that mismatch with several incompatible conventions. transfer in the evaluated settings, not a claim that every pooled model outperformed every single-embodiment baseline.

Data quality is a separate question from whether a particular pooled-training experiment shows transfer. In his October 2025 post, Moritz Reuss writes that “OXE is mostly low-quality data” and that “we still lack good methods to quantify data quality in imitation learning.” He presents the post as his personal selection and opinion. This is Reuss’s critique, not a demonstrated normalization artifact or a statement of community consensus. Reuss 2025↓State of VLA Research at ICLR 2026Moritz Reuss, 2025

Octo and OpenVLA: the open models

Two 2024 releases made the VLA recipe reproducible outside Google.

Octo is a generalist robot policy with released 27M- and 93M-parameter checkpoints, trained on 800k Open X-Embodiment trajectories. Its language tokenizer separately uses pretrained T5-base (111M parameters). Observation and task tokens enter a block-wise-masked transformer; learned readout tokens attend to preceding inputs but do not influence those inputs through attention. A lightweight diffusion head maps the readouts to action chunks. Adding a sensor or output specification during fine-tuning can require new positional embeddings, a lightweight encoder or a new head, while retaining the transformer weights. It is not a promise that every new input requires no architectural work. Octo Model Team 2024↓Octo: An Open-Source Generalist Robot PolicyOcto Model Team, Dibya Ghosh, Homer Walke et al., 2024

In its zero-shot WidowX evaluation, the paper reports a “25% higher success rate” with goal-image conditioning than with language conditioning. That is the paper’s wording; it must not be silently converted into a 25-percentage-point difference, since the paragraph does not give the underlying paired rates or explicitly label percentage points. The authors suggest that goal images convey more task information and identify better language conditioning as future work; they also note that only 56% of their pretraining data has language annotations. Octo Model Team 2024↓Octo: An Open-Source Generalist Robot PolicyOcto Model Team, Dibya Ghosh, Homer Walke et al., 2024

OpenVLA builds on the Prismatic vision-language architecture with Llama 2 7B and fused DINOv2/SigLIP visual features. It is trained on 970k robot demonstrations and predicts seven action coordinates: position control, orientation control and gripper control. For each coordinate, 256 bins uniformly divide the interval between the training data’s 1st and 99th quantiles, rather than the observed minimum and maximum. Bin indices replace the 256 least-used Llama vocabulary entries and are predicted autoregressively. Kim 2024↓OpenVLA: An Open-Source Vision-Language-Action ModelMoo Jin Kim, Karl Pertsch, Siddharth Karamcheti et al., 2024 Kim 2025↓Fine-Tuning Vision-Language-Action Models: Optimizing Speed and SuccessMoo Jin Kim, Chelsea Finn, Percy Liang, 2025

The v3 paper’s headline comparison reports a 16.5-percentage-point advantage in absolute task success over RT-2-X (55B), using the authors’ “7x fewer parameters” description. This is a comparison across 29 tasks on two embodiments: 17 WidowX tasks with ten trials each and 12 Google-robot tasks with five trials each, evaluated with matched initial robot and object states. It is not a general ranking across all robot tasks; the task definitions include partial credit in some cases. Kim 2024↓OpenVLA: An Open-Source Vision-Language-Action ModelMoo Jin Kim, Karl Pertsch, Siddharth Karamcheti et al., 2024

The discrete-token throughput problem

OpenVLA’s original formulation predicts seven action tokens sequentially per timestep. That serial decoding is an inference cost, not a universal robot-control frequency. Kim 2025↓Fine-Tuning Vision-Language-Action Models: Optimizing Speed and SuccessMoo Jin Kim, Chelsea Finn, Percy Liang, 2025 RT-2’s authors report 1 to 3 Hz for the 55B PaLI-X variant and around 5 Hz for its 5B variant when querying their multi-TPU cloud service over the network. Brohan 2023↓RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic ControlAnthony Brohan, Noah Brown, Justice Carbajal et al., 2023 OpenVLA reports approximately 6 Hz inference on one RTX 4090 in bfloat16, without compilation or speculative decoding. Its separate Franka fine-tuning setups use 5 Hz and 15 Hz non-blocking controllers, and its quantization analysis explicitly discusses how slower inference changes controller dynamics. Do not substitute the 4090 inference figure for an executed rate on every robot. Kim 2024↓OpenVLA: An Open-Source Vision-Language-Action ModelMoo Jin Kim, Karl Pertsch, Siddharth Karamcheti et al., 2024

Kim 2025↓Fine-Tuning Vision-Language-Action Models: Optimizing Speed and SuccessMoo Jin Kim, Chelsea Finn, Percy Liang, 2025 OpenVLA-OFT (2025) studies how to adapt OpenVLA, rather than showing that representation alone determines policy quality. Its recipe combines bidirectional parallel decoding, action chunkingaction chunkingPredicting a sequence of future actions in one inference instead of a single action per timestep. Introduced with ACT on the ALOHA platform, chunking cuts the number of sequential decisions in an episode by the chunk length, which reduces compounding error and gives the policy the temporal context it needs to represent pauses, at the cost of committing to a plan between replans., continuous actions, and an L1-trained MLP action head. The ALOHA variant, OpenVLA-OFT+, also uses FiLM to strengthen language grounding.

The paper's LIBERO table reports 76.5% average success for the original fine-tuning recipe and 97.1% for OFT with an added wrist image and proprioceptive state. The average covers four ten-task suites; OFT is fine-tuned separately per suite, with 500 evaluation episodes per suite. The headline throughput result is a different input setting: on an A100, averaged over 100 one-image queries, L1 OFT generates 109.7 actions/s versus 4.2 for OpenVLA, about 26 times as many. With the added inputs used for the 97.1% result, throughput is 71.4 actions/s. These are action-generation rates, not robot-control frequencies. Kim 2025↓Fine-Tuning Vision-Language-Action Models: Optimizing Speed and SuccessMoo Jin Kim, Chelsea Finn, Percy Liang, 2025

The real ALOHA experiments use three images, robot state, and absolute joint targets at 25 Hz, reduced from the platform's original 50 Hz. OFT+ predicts and executes all 25 actions before replanning; LIBERO predicts and executes eight. The ALOHA tasks include folding clothes and language-directed scooping and object placement. They use partial-task-completion scoring, not the LIBERO binary-success protocol. Kim 2025↓Fine-Tuning Vision-Language-Action Models: Optimizing Speed and SuccessMoo Jin Kim, Chelsea Finn, Percy Liang, 2025

In ACT’s chunk-size ablation, the authors train separate policies for each chunk size with temporal ensembling disabled. Results average four settings: two simulated tasks, each with human or scripted demonstrations. The ACT curve rises from 1% at k=1 to 44% at k=100; those numbers are not the BC-ConvMLP baseline. Adding chunks also improves BC-ConvMLP and VINN in these settings. A separate ablation finds that removing ACT’s CVAE objective barely changes scripted-data results but lowers human-data success from 35.3% to 2%. Neither experiment establishes that probabilistic modeling is generally unnecessary. Zhao 2023↓Learning Fine-Grained Bimanual Manipulation with Low-Cost HardwareTony Z. Zhao, Vikash Kumar, Sergey Levine et al., RSS 2023

Where this leaves the field

These studies demonstrate particular transfers from web pretraining and pooled robot data, not universal improvement from scale alone. Their action interfaces, data mixtures, input modalities and execution protocols differ. Read the reported token-generation rates and robot-control rates separately, and compare results only with their task and evaluation settings attached. Brohan 2023↓RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic ControlAnthony Brohan, Noah Brown, Justice Carbajal et al., 2023 Kim 2024↓OpenVLA: An Open-Source Vision-Language-Action ModelMoo Jin Kim, Karl Pertsch, Siddharth Karamcheti et al., 2024

A 2026 selection rule

Treat “VLA” as two decisions, not one model category. The first is the prior: which visual, language and robot experience the backbone brings to the task. The second is the control interface: what action the model emits, over what horizon, in which frame and at what measured rate. A large semantic prior can improve instruction following while a poor action interface still makes the system unusable at contact. OpenVLA-OFT is the clean example because changing the decoding and action head materially changed both throughput and success without replacing the underlying VLA Kim 2025↓Fine-Tuning Vision-Language-Action Models: Optimizing Speed and SuccessMoo Jin Kim, Chelsea Finn, Percy Liang, 2025.

Before choosing a checkpoint, write four rows: required semantic novelty, available embodiment data, action contract and latency budget. Use the foundation-model guide to select the prior family, then the action-space guide to specify the output. Only compare models after those two contracts match. Parameter count and benchmark average are weak selectors when one candidate cannot express the controller command or meet its deadline.