Generative Video World Models

The third paradigm in the taxonomy is the one with the money and the attention: predict future pixels given the current frame and an action or a text prompt, then use the generator as a stand-in for the environment. A model that does this well is a learned simulator you can query with candidate actions, which is why the same systems show up under four different agendas: generating training data, evaluating policies, running RL post-training, and powering interactive worlds.

Whether any of them is a simulator rather than a movie generator comes down to one property, and it is measurable. Toggle the conditioning strength below from strong to weak: two rollouts from one shared frame under different actions collapse onto the same future, the action-sensitivity score falls to near zero, and the visual-realism score does not move at all.

Cosmos 3: the omni-model

In the Cosmos 3 report, NVIDIA argues that a hypothetical home robot would otherwise need a VLM to plan, a VLA or WAM to generate actions, and a forward-dynamics model to simulate and evaluate future states. It calls that arrangement "fragmented" and "computationally wasteful" NVIDIA 2026↓Cosmos 3: Omnimodal World Models for Physical AINVIDIA, Aditi, Niket Agarwal et al., 2026. Cosmos 3, launched at GTC Taipei in late May 2026, unifies those capabilities in one omni-model NVIDIA 2026↓NVIDIA Launches Cosmos 3, the Open Frontier Foundation Model for Physical AINVIDIA, NVIDIA Newsroom, 2026. Cosmos 3 uses a Mixture-of-Transformers backbone. Modality-specific encoders project inputs into a shared representation space. Its autoregressive subsequence handles language tokens and ViT-encoded images and video; its diffusion subsequence handles VAE-encoded images and video, audio, and action tokens. Each transformer layer has separate reasoner and generator parameters. Autoregressive tokens attend causally only within their own subsequence, while diffusion tokens attend to both subsequences; the reasoning stream is not updated from diffusion tokens NVIDIA 2026↓Cosmos 3: Omnimodal World Models for Physical AINVIDIA, Aditi, Niket Agarwal et al., 2026.

Section 4.1.1 trains the reasoner by next-token prediction. Section 4.2 specifies rectified flow matching for the generator, whereas the infrastructure discussion in Section 5.2 calls the generator loss "EDM loss". Those two descriptions are not treated as equivalent here NVIDIA 2026↓Cosmos 3: Omnimodal World Models for Physical AINVIDIA, Aditi, Niket Agarwal et al., 2026.

The common architecture supports VLMvision-language modelA model pretrained jointly on web-scale image and text data, so visual recognition and language semantics live in one set of weights. RT-2's bet was that this pretraining is an asset for robot control: co-fine-tuning a vision-language model on robot trajectories and its original web data together transfers semantic knowledge, recognizing objects and following instructions the robot data never covered, into the policy. The vision-language-action models that followed all start from such a backbone. understanding, video generation, forward dynamics, inverse dynamics, and joint video-action policy mode. That is not evidence that one unchanged checkpoint supplies every specialized result: the report separately post-trains image and video generators and a DROID robot policy. The DROID pilot starts from mid-trained Cosmos3-Nano and freshly initializes the action encoder, action-decoding MLP, and action embeddings NVIDIA 2026↓Cosmos 3: Omnimodal World Models for Physical AINVIDIA, Aditi, Niket Agarwal et al., 2026.

Section 2.2.2 describes three action modes. Forward dynamics denoises future video conditioned on observed context and clean actions. Inverse dynamics denoises actions from clean observed video transitions. Policy mode jointly denoises future video and actions given observed context. These modes use the shared language prefix; Figure 4 omits language and special tokens for compactness NVIDIA 2026↓Cosmos 3: Omnimodal World Models for Physical AINVIDIA, Aditi, Niket Agarwal et al., 2026. Section 2.5 reports Cosmos 3 Nano at 16B, built upon a dense 8B transformer, and Cosmos 3 Super at 64B, built upon a dense 32B transformer. Both use the report's dual-tower MoT architecture NVIDIA 2026↓Cosmos 3: Omnimodal World Models for Physical AINVIDIA, Aditi, Niket Agarwal et al., 2026.

Genie 2 and Genie 3: interactive worlds with an honest limitation list

DeepMind's Genie line approaches from play rather than from robots. The original Genie learned latent actions from video-game footage and generated 2D worlds at about one frame per second Bruce 2024↓Genie: Generative Interactive EnvironmentsJake Bruce, Michael Dennis, Ashley Edwards et al., 2024. Genie 2 (December 2024) moved to 3D, with consistent worlds for up to a minute and most examples lasting 10 to 20 seconds Parker-Holder 2024↓Genie 2: A large-scale foundation world modelJack Parker-Holder, Philip Ball, Jake Bruce et al., Google DeepMind, 2024. In their August 5, 2025 announcement, Jack Parker-Holder and Shlomi Fruchter describe Genie 3 as a limited research preview. They report text-prompted worlds navigable in real time at 24 fps and 720p, remaining largely consistent for a few minutes Parker-Holder 2025↓Genie 3: A new frontier for world modelsJack Parker-Holder, Shlomi Fruchter, Google DeepMind, 2025. Promptable world events can alter weather or introduce objects and characters through text input; DeepMind presents these interventions as a way to broaden counterfactual scenarios. To test compatibility with future agent training, DeepMind gave a recent version of SIMA goals in generated worlds. SIMA sent navigation actions to Genie 3, while the world modelworld modelA model of how an environment evolves that an agent can query to make decisions: given the current observation or state and a candidate action, it predicts what happens next, so a policy can be trained, evaluated, or planned against the model instead of the real world. The 2026 robotics survey draws the functional line: producing plausible future images is not enough, because a system qualifies only if its predictions change under the agent's action in a way that is useful for decision-making. The single name covers at least six architecturally distinct paradigms, from compact latent dynamics models to action-conditioned video generators. received no agent goal and simulated the future from those actions. This is a navigation demonstration, not a reported robot-manipulation training result Parker-Holder 2025↓Genie 3: A new frontier for world modelsJack Parker-Holder, Shlomi Fruchter, Google DeepMind, 2025.

DeepMind lists the following limitations of Genie 3 Parker-Holder 2025↓Genie 3: A new frontier for world modelsJack Parker-Holder, Shlomi Fruchter, Google DeepMind, 2025:

• Limited action space. Promptable world events allow environmental interventions that are not necessarily performed by the agent itself. Direct agent actions remain constrained.
• Other agents. Accurately modelling complex interactions between multiple independent agents in shared environments remains a research challenge.
• Geographic accuracy. The model cannot simulate real-world locations with perfect geographic accuracy.
• Text rendering. Clear, legible text is often only generated when it is supplied in the input world description.
• Limited interaction duration. The model supports a few minutes of continuous interaction, rather than extended hours.

Deployment makes the duration constraint concrete. Project Genie, the consumer interface released in January 2026 for AI Ultra subscribers, caps sessions at 60 seconds per world Whitwam 2026↓Google Project Genie lets you create interactive worlds from a photo or promptRyan Whitwam, Ars Technica, 2026. Real time at 24 fps does not mean affordable at length.

GR-1 and GR-2: video pre-training pointed at manipulation

The robotics-specific lineage started with GR-1, which established the pattern: large-scale video generative pre-training, then fine-tuning for multi-task language-conditioned manipulation, predicting future images and actions end to end. On CALVIN it moved the success rate from 88.9% to 94.9%, and in zero-shot unseen scenes from 53.3% to 85.4% Wu 2023↓Unleashing Large-Scale Video Generative Pre-training for Visual Robot ManipulationHongtao Wu, Ya Jing, Chilam Cheang et al., 2023. GR-2 scaled the same idea: pre-training on 38 million internet video clips and over 50 billion tokens to capture world dynamics, then joint fine-tuning for video generation and action prediction on robot trajectories, with a reported 97.7% average success rate across more than 100 tasks Cheang 2024↓GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot ManipulationChi-Lam Cheang, Guangzeng Chen, Ya Jing et al., 2024. In both systems the video prediction is not decorative. It is the mechanism that imports web-scale dynamics into a manipulation policy.

1X and Odyssey: the commercial edges

1X treats its world model as the route to self-learning humanoids. The company credits the model with letting NEO generalize to unseen tasks with zero-shot execution, and in June 2026 it launched a dedicated World Model Lab, led by Sam Sinha (previously a founding research scientist at Luma AI), to pre-train video foundation models on a mixture of web video, egocentric human data, simulation, teleoperated robot data, and on-policy NEO data 1X 2026↓1X Launches World Model Lab to Scale Humanoid Intelligence1X, 2026.

Odyssey is the clearest consumer expression of the paradigm. Odyssey-2 (October 2025) is a causal, autoregressive interactive video model: each frame is generated only from prior frames and user input, streamed in real time at a new frame every 50 ms (about 20 fps), and steerable with text as it plays Cameron 2025↓Introducing Odyssey-2: A General-Purpose World ModelOliver Cameron, Odyssey, 2025. The follow-ons push the same direction, with Starchild-1 adding audio alongside video Cameron 2026↓Starchild-1: The First Real-Time Multimodal World ModelOliver Cameron, Odyssey, 2026 and Agora-1 letting four participants share one simulation Cameron 2026↓Agora-1: The Multi-Agent World ModelOliver Cameron, Odyssey, 2026.

The conditioning problem

A world model is useful for control only if its predictions change under the candidate action in a way that tracks physical consequence. The 2026 survey names weak action conditioning the top open challenge: many models are trained mostly from observation history and task intent, so their futures are "semantically plausible or intention-consistent, but not necessarily faithful to the physical consequences of the candidate action" Hou 2026↓World Model for Robot Learning: A Comprehensive SurveyBohan Hou, Gen Li, Jindou Jia et al., 2026.

The failure is invisible in a demo reel, because a weakly conditioned model still renders sharp, realistic video. It just renders the same plausible future no matter which action you issue. That is what the figure at the top of this module measures: realism and action sensitivity are decoupled quantities, and only one of them tells you whether the model heard the action.

The SIMA demonstration described above tests navigation in generated worlds; it does not establish manipulation performance Parker-Holder 2025↓Genie 3: A new frontier for world modelsJack Parker-Holder, Shlomi Fruchter, Google DeepMind, 2025.

Does it work as a simulator? The evidence, both ways

The Interactive World Simulator uses a CNN image encoder, a consistency-model image decoder and an action-conditioned latent dynamicslatent dynamicsA world-model paradigm that predicts in a compact learned latent space instead of in pixels: the model carries a recurrent latent state, forecasts how that state and the reward evolve under candidate actions, and the agent learns or plans entirely inside the imagined rollouts. Dreamer established the recipe of training an actor-critic purely on imagined trajectories, and TD-MPC2 showed the image decoder can be dropped altogether, scoring candidate action sequences under the latent model with a learned terminal value instead of reconstructing pixels. model. The v1 paper reports more than 10 minutes of visually stable generated-video interaction at up to 15 FPS on a single RTX 4090. Its default experimental image resolution is 128×128, while quantitative video comparisons use 192 steps (19.2 seconds), not the ten-minute horizon. These are video-generation results, not robot-control frequencies or a guarantee of physically accurate dynamics on every task Wang 2026↓Interactive World Simulator for Robot Policy Training and EvaluationYixuan Wang, Rhythm Syed, Fangyu Wu et al., 2026.

IWS also studies policy training with 100-episode mixtures of real and world-model-generated demonstrations. Its reported average task scores for generated-only versus real-only policy data are 87.9% versus 90.3% for DP and 76.2% versus 73.6% for ACT; the same discussion reports π0.5 improving from 73.1% to 88.8% as real data increases. This is source-specific comparability, not equivalence for every policy or task. Each mixture point uses 100 MuJoCo evaluations or 10 physical evaluations, and the simulator itself was trained on interaction data, including real play data: “generated-only” does not mean zero real data overall. A separate four-task policy-evaluation experiment samples 20 initial configurations from the world-model training distribution and reports positively biased simulator scores except for T pushing Wang 2026↓Interactive World Simulator for Robot Policy Training and EvaluationYixuan Wang, Rhythm Syed, Fangyu Wu et al., 2026.

RoboWorld combines an autoregressive video world model with a task-progress-aware VLM scorer. Step Forcing trains on self-forwarded denoising priors while retaining data-grounded anchor contexts; four-step denoising and contextual KV caching support fast inference. In v4, 4,186 generated 30-second closed-loop rollouts yield eight policy-level aggregate scores compared with the 26 February 2026 RoboArena leaderboard: Pearson r = 0.989 and Spearman rho = 0.970, with p < 0.001 reported for both. GPT-4o scores task progress from 0 to 5, using fixed external views for success and the wrist view only to identify world-model errors. The result is neither a per-task correlation nor 4,186 independent correlation points. It is also not calibrated success probability or absolute agreement: the paper reports contact-induced object artifacts and VLM scores about one point above human scores in approximately 100 sampled rollouts. Its separate 256-trajectory held-out open-loop video test does not establish an independent tuning/test split for the headline closed-loop correlation Jeon 2026↓RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy EvaluationByeongguk Jeon, Seonghyeon Ye, JaeHyeok Doo et al., 2026.

The negative evidence is structural. The survey's verdict on evaluation is blunt: visual plausibility is only a weak proxy for control utility, and action-grounded consistency and controllability are much more reliable indicators of downstream usefulness Hou 2026↓World Model for Robot Learning: A Comprehensive SurveyBohan Hou, Gen Li, Jindou Jia et al., 2026. Beyond weak conditioning, the survey lists missing modalities (vision and proprioception cannot capture friction, stiffness, or contact stability, and it calls tactile integration indispensable) and cost (world-model-based policies are far more computationally intensive than VLAsvision-language-action modelA robot policy built by adapting a vision-language model, pretrained on web-scale image and text data, to output robot actions. RT-2 established the approach by representing actions as text tokens and co-fine-tuning on web and robot data together, showing that semantic knowledge from web pretraining transfers into control. The same recipe underlies OpenVLA and the π series of generalist policies., with diffusion denoising latency the specific culprit) Hou 2026↓World Model for Robot Learning: A Comprehensive SurveyBohan Hou, Gen Li, Jindou Jia et al., 2026. Every positive result above also sits on quasi-static tabletop tasks. Nothing published shows a generative video model producing usable training signal for contact-rich assembly or locomotion.

The net assessment, mid-2026: generative video world models are demonstrably useful today for policy evaluation, for synthetic demonstration generation, and for RL post-training in narrow, in-distribution task families. They are not a general replacement for a physics engine, and nobody credible claims they are. The gating property is the one in the interactive: whether the predicted future actually listens to the action.

An evaluation ladder for video models

Begin below photorealism. Run paired interventions in which the initial frames are identical and only the action changes. Verify direction, contact onset and object identity before measuring image quality. Next test one-step prediction on held-out physical transitions, then increase the horizon and record where geometry, action sensitivity or temporal consistency fails. A long attractive rollout does not compensate for an incorrect first contact.

Only after those checks should the model rank candidate policies or generate training data. Compare its ranking with blinded real-robot trials and report uncertainty by task family. For synthetic data, train the same policy with and without generated episodes, then evaluate both on untouched real scenes. Preserve prompts, action traces, seeds and model versions so a result can be reproduced. The world-model evaluation guide gives the full protocol, and world models versus simulators explains when an engine or hybrid system is the better evidence source.