A world model is a model that predicts how an environment evolves in a form useful for a robot's decisions. Hou and colleagues' 2026 survey applies this functional definition to visual, latent, structured and symbolic predictions.Source 1

Two rows of frames of a simulated four-legged robot, labelled True and Model: two context frames, then open-loop predictions from step 10 to step 50 in which the model row closely tracks the true row.
A world model predicts how an environment evolves under actions. From 5 context frames and the actions alone, DreamerV3 predicts 45 frames of a simulated quadruped; the model row tracks the true row.
Paper figure: Danijar Hafner et al. / arXiv. Licence: CC BY 4.0.

Ha and Schmidhuber's 2018 world model learned "a compressed spatial and temporal representation of the environment"Source 13. Switch "What it imagines" between pictures, a compressed summary and plain facts; "Adjust more" lists six example groups. The six groups are this article's authored selection, not an exhaustive or universally agreed scientific taxonomy.

Six kinds of world model
Robot imaginations: some predict pictures, others compact summaries
What it imagines
Adjust more
Used for
Policy learning
Not used for
Planning
Evaluation
Data generation
NowNext action:pick up the cupThe next pictureA compressed summaryPlain factscup in gripperThis model imagines only the summary;it draws pictures only during training

Illustrative schematic: drawn by hand, not a model's output.

Selected: Latent dynamics. Imagines a compressed summary. Predicts: Next latent state + reward + continuation flag. Latent-dynamics world model (Dreamer-style), used for policy learning.

To plan, a robot imagines what happens next; models differ in whether they imagine full pictures, a compressed summary, or plain facts.
How this was made

On the left is what the robot sees now: an arm above a cup on a table, about to pick the cup up. On the right are the three forms an imagined next moment can take: the next picture itself, a compressed summary (a short list of numbers, drawn as beads), or plain facts about the objects. The marked form is the one the selected group predicts; “Adjust more” lists the six groups and their uses.

What each group predicts and what it is used for:

Latent-dynamics world model (Dreamer-style): Next latent state + reward + continuation flag. Used for policy learning.
Decoder-free latent model for planning (TD-MPC-style): Next latent state + reward; no reconstruction, no decoder. Used for policy learning and planning.
Action-conditioned generative video model: Future pixels conditioned on the current frame(s) + action/text. Used for policy learning, evaluation and data generation.
Non-generative joint-embedding predictor (JEPA): Future representation, never pixels. Used for planning.
Unified world-action model: Future frames and action chunks from one backbone. Used for policy learning.
Symbolic / structured world model: Transitions over predicates, object relations, occupancy. Used for planning.

Dreamer-style models also learn a decoder that turns a summary back into a picture, but they use it only as a training signal; TD-MPC-style and JEPA models have no decoder at all. The uses are this article's reading of each group against the survey's four: policy learning, planning, evaluation and data generation.

The six groups are this article's selection, not an exhaustive taxonomy. The arm, the cup, the beads and the facts are drawn by hand to show the form each prediction takes; none of them is a model's output.

Latent dynamics (Dreamer-style), one of 3 groups out of 6 that imagine a compressed summary, predicts the next latent, a reward and a continue signal, and draws pictures only during training; of the 4 uses, it serves policy learning.

Current world-model group
group
Latent dynamics
imagines
a compressed summary
predicts
Next latent state + reward + continuation flag
used for
1 of 4: policy learning
Original schematic; the drawings are illustrative.
6
example groups
selected in this article
150+
DreamerV3 tasks
one fixed hyperparameter set
317M
TD-MPC2 params
one offline agent, 80 tasks
>1M
V-JEPA 2 video
video hours; action-free pretraining

A model counts only if it predicts the effect of actions

Hou and colleagues count a model only if it predicts how the future changes under robot-relevant actions and supports embodied decision-making. In their words, "a model does not qualify as a world model in our sense simply because it generates plausible future images or videos". The survey names weak action conditioning a technical bottleneck, since it allows futures with no causal tie to the pending actionSource 1. A second 2026 survey likewise defines a world model as "an action-conditioned predictive system"Source 14.

Six example groups

Six example groups selected for this article
ParadigmWhat it predictsIn what spaceTrained onPrimary useRepresentative systems
Latent-dynamics world model (Dreamer-style)Next latent state + reward + continuation flagLearned compact latent (stochastic + deterministic); decoded to pixels during trainingThe agent's own interaction dataPolicy optimization on imagined rollouts; sample-efficient RLDreamerV3, DayDreamer, Robotic World Model
Decoder-free latent model for planning (TD-MPC-style)Next latent state + reward; no reconstruction, no decoderImplicit latent trained only for value/reward predictionInteraction dataLatent-space planning: MPPI or policy-seeded gradient ascentTD-MPC, TD-MPC2, Dream-MPC
Action-conditioned generative video modelFuture pixels conditioned on the current frame(s) + action/textPixel or VAE-latent video spaceInternet video + robot trajectoriesData generation, policy evaluation, RL post-training, interactive worldsCosmos 3, Genie 3, Odyssey, Interactive World Simulator
Non-generative joint-embedding predictor (JEPA)Future representation, never pixelsLearned embedding space; masked/future embedding predictionInternet video + a small action-labeled robot setZero-shot planning by energy minimization over latent goalsV-JEPA, V-JEPA 2, V-JEPA 2-AC
Unified world-action modelFuture frames and action chunks from one backboneShared backbone; parallel generative and action headsRobot trajectories + videoPolicy with world-modeling as auxiliary objective / implicit lookaheadCosmos Policy, WorldVLA, DreamZero
Symbolic / structured world modelTransitions over predicates, object relations, occupancyDiscrete/relational or 3D-occupancy spaceCurated or perception-grounded dataLong-horizon task planning without pixel-space error accumulationOccWorld, symbolic-abstraction hybrids

Latent dynamics: predict a compact state, dream in it

DreamerV3 uses a recurrent state-space model with stochastic latents and reconstructs inputs; its actor and critic learn from imagined trajectories, and the critic also receives a loss on replay-buffer trajectories. It ran more than 150 tasks in eight domains with fixed hyperparametersSource 2. Hafner and colleagues' 2025 Nature article calls it, to their knowledge, the first algorithm to collect Minecraft diamonds from scratch without human data or adaptive curriculaSource 15, in an environment with abstract crafting actions, accelerated block breaking and sparse intermediate rewardsSource 2. See the latent-dynamics explanation.

Dreamer 4 draws most knowledge from unlabeled video, needing little action-labeled dataSource 16. DayDreamer taught a Unitree A1 to roll off its back, stand and walk in about one hour of real training with no simulator or reset policy, though a PD controller and manual repositioning remainedSource 4.

Decoder-free latent: no image, just value

TD-MPC2 learns joint-embedding, reward and value predictions without reconstructing observations, and its MPPI planner scores sampled action sequences by latent rollouts plus a terminal value. Its 317M-parameter model trained offline on 545M transitions from 240 single-task agents, a run distinct from the 104-task online benchmarkSource 3.

Generative video: predict the pixels

These models predict future pixels from frames plus an action or text condition; see generative video. NVIDIA presents Cosmos 3 as one Mixture-of-Transformers for understanding, generation, policy and forward and inverse dynamicsSource 5. DeepMind reports that Genie 3 renders text-prompted worlds navigable in real time at 24 fps and 720p, largely consistent for a few minutes, with constrained agent actions among its limitsSource 6.

JEPA: predict the representation, never the pixels

V-JEPA trains "solely using a feature prediction objective"Source 17. V-JEPA 2 pretrains action-free on over a million hours of video plus images. V-JEPA 2-AC freezes the encoder and trains an action-conditioned predictor on under 62 hours of filtered left-camera Droid video, which carries no reward, task-type or success labels but includes Franka Panda teleoperationSource 7.

The same weights and code ran zero-shot on Franka Panda arms in two labs absent from Droid, with hand-chosen cameras and, for pick-and-place, two subgoal imagesSource 7. See the JEPA module.

Unified world-action: one backbone, two heads

A world-action model (WAM) uses a pretrained video backbone to predict both future states and robot actionsSource 18; unlike a VLA, it learns physical dynamicsSource 19. WorldVLA interleaves action and image tokens and reports that the objectives improve each otherSource 8, and Cosmos Policy post-trains Cosmos-Predict2 into a policy with no architectural modificationsSource 9. NVIDIA reports that DreamZero runs a 14B video-diffusion model as a 7 Hz closed-loop policy with over twice the generalization of state-of-the-art VLAs on real robotsSource 19. Fast-WAM stays competitive while skipping test-time future generation, and removing video co-training hurts far moreSource 20.

Two rows of ALOHA robot frames with arrows between them: putting an object on a plate, folding a shirt, putting candies in a bowl and putting candy in a zip bag.
Cosmos Policy fine-tunes a pretrained video model into a robot policy, so a predictive model becomes the controller. Rollouts on four ALOHA tasks, from folding a shirt to putting candy in a zip bag.
Paper figure: Moo Jin Kim et al. / arXiv. Licence: CC BY 4.0.

Symbolic and structured: predict predicates

These models predict transitions over predicates, object relations or 3D occupancy; OccWorld, from autonomous driving, predicts how scene occupancy evolves and plans competitively without instance or map supervisionSource 10. PointWorld predicts the 3D point flows that a sequence of robot actions produces, and at 0.1 s per inference it drives model-predictive control on a real Franka arm with no demonstrationsSource 30. Silver and colleagues treat learned operators as "a lossy abstraction of the transition model"Source 21, and VisualPredicator, which invents its predicates online, generalizes better out of distributionSource 22.

Two boundary cases

A physics engine is a world model

MuJoCo is an explicit, hand-specified state-transition model: given a state and an action, it predicts the next stateSource 11. TD-MPC2 and classical MPC therefore share one pattern, with learned or derived dynamics.

A 3D Gaussian Splatting twin is not

A 3DGS reconstruction is a learned renderer that predicts a scene from a new camera poseSource 12; its dynamics come from the physics engine underneath, so alone it fails the functional cut. PhysGaussian adds dynamics by evolving the Gaussians with a Material Point Method solverSource 23, and World Labs' Marble exports splats with collider meshes "intended for coarse physics simulation"Source 24. See generative simulation.

A photorealistic rendered view of a garden: a round wooden table with a vase on it, a paved patio, shrubs and a brick wall.
A 3D Gaussian Splatting reconstruction is a learned renderer: it predicts a real scene from a new camera pose by rasterising the fitted Gaussians. A garden scene rendered by the original method.
Paper figure: Bernhard Kerbl et al. / arXiv. Licence: CC BY 4.0.

Video models serve evaluation and data; latent models plan without images

Google DeepMind reports a Veo-based simulator predicted policies' relative performance, checked against 1,600+ real evaluations of eight Gemini Robotics checkpointsSource 25. The Interactive World Simulator sustains over 10 minutes of interaction at 15 FPS on one RTX 4090, and policies trained on its data matched those trained on as much real dataSource 26. Decoder-free latent and JEPA models plan without showing an image.

Three tests before believing a world-model claim

The intervention test varies the action with history fixed, since "action-marginalized" hallucination is a measured failure modeSource 27. The ranking test asks whether the model orders policies as reality does. The transfer test repeats both across objects, contacts, viewpoints and horizons: in controlled scaling experiments, video models failed out of distribution and showed "case-based" generalizationSource 28. Visual quality counts last, since in six major video generators physical understanding was "severely limited, and unrelated to visual realism"Source 29.

Hou and colleagues separate learned simulation for reinforcement learning, rollout evaluation and video generation for supervisionSource 1. The evaluation guide, model-based robot learning and world models versus simulators go further.

Self-check

A vendor pitches a photorealistic 3D Gaussian Splatting reconstruction of your workcell as a world model for training a manipulation policy. What is the honest engineering verdict?
Read the reasoning
  • It qualifies: it predicts the scene from any camera pose, which is a predictionThis is the most common misreading of the real-to-sim literature. A 3DGS twin is a learned renderer: it predicts appearance from a new viewpoint and has no answer to what happens under an action. The survey’s functional cut is explicit on this: generating plausible future images alone does not qualify a model as a world model.
  • It is a learned renderer bolted onto a physics engine: appearance is learned, dynamics are notThis is the correct division of labor. The splat supplies appearance while a conventional physics engine still supplies dynamics, and the reconstruction freezes the scene it captured: articulated objects, deformables, and lighting changes each need extra machinery on top.
  • It fails the cut because nothing hand-specified counts as a world modelOnly action-dependence determines the cut; learned dynamics are not required. MuJoCo is an explicit, hand-specified state-transition model and it qualifies: given a state and an action it predicts the next state. The 3DGS twin fails for having no action answer, not for lacking weights.

Ask what happens under an action: a renderer that only answers where the camera goes fails the functional cut, whatever the video looks like.