World Models vs Physics Simulators

A physics simulator predicts the next state from explicit geometry, dynamics and contact rules. A learned world modelworld modelA model of how an environment evolves that an agent can query to make decisions: given the current observation or state and a candidate action, it predicts what happens next, so a policy can be trained, evaluated, or planned against the model instead of the real world. The 2026 robotics survey draws the functional line: producing plausible future images is not enough, because a system qualifies only if its predictions change under the agent's action in a way that is useful for decision-making. The single name covers at least six architecturally distinct paradigms, from compact latent dynamics models to action-conditioned video generators. predicts from data. Both are models of what happens under an action; neither is automatically faithful to the real robot. The practical choice is which errors are measurable, which counterfactuals must be queried, and how far beyond the training distribution the system will operate.

The most productive systems increasingly combine them. Physics supplies structure and broad counterfactual coverage. Learning supplies appearance, actuator residuals, contacts or task distributions that are expensive to specify.

Try the check below and choose the actuator-residual option that preserves the strongest known structure for a contact-rich training task.

Self-check

You have accurate arm geometry and rough rigid-body physics, but the simulated actuator response differs systematically from hardware. What is the most data-efficient first correction?

Replace the entire simulator with an unconstrained video world modelLearn an actuator residual while retaining explicit geometry and physicsRandomize only the rendered textures

Read the reasoning

• Replace the entire simulator with an unconstrained video world modelThis discards useful geometry and asks limited robot data to relearn every transition. It also makes forces and constraints harder to inspect.
• Learn an actuator residual while retaining explicit geometry and physicsThis targets the measured mismatch and keeps known structure intact. The residual can be validated directly against held-out hardware trajectories.
• Randomize only the rendered texturesTexture randomization can improve visual transfer, but it does not correct a systematic actuator-dynamics error.

Keep trustworthy structure explicit and spend learned capacity on the residual you can measure.

They solve different specification problems

MuJoCo models articulated bodies, constraints and contact with explicit equations and parameters Todorov 2012↓MuJoCo: A physics engine for model-based controlEmanuel Todorov, Tom Erez, Yuval Tassa, IROS 2012. Isaac Lab builds a robot-learning workflow around GPU-accelerated simulation, sensors, task definitions and large batches of environments NVIDIA 2025↓Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot LearningNVIDIA, Mayank Mittal, Yunrong Guo et al., 2025. These systems are strongest when the relevant objects and rules can be represented and their parameters can be estimated.

A learned world model avoids writing every rule. It estimates transitions from observations and actions, often in a latent or video space. This is attractive for complex appearance, deformable interaction and real environments whose full state is unavailable. It also replaces parameter error with dataset and generalization error.

The current survey of world models for robot learning treats learned simulators as one role among planning, policy learning, evaluation and data generation Hou 2026↓World Model for Robot Learning: A Comprehensive SurveyBohan Hou, Gen Li, Jindou Jia et al., 2026. That role should not erase the distinction between a simulator whose state variables are explicit and a generator whose internal state must be inferred through outputs.

Comparison by engineering property

The table is not a winner. It identifies where each system makes uncertainty visible.

What each representation costs to run

A world model's usefulness is bounded by what it costs to query. The table below compares four deployed forms of world knowledge on a single machine: a text LLM as a familiar decode reference, an action-conditioned latent world model, an autoregressive or diffusion video world model, and an explicit 3D scene. The figures come from the cited papers and vendor documentation; the FLOP row is a derived order-of-magnitude estimate, not a measurement.

The LLM column anchors the scale. An 8B dense decoder such as Llama 3 8B keeps about 16 GB of weights in fp16 and emits a token in tens of milliseconds on a consumer card; at that size, decode is bound by reading weights rather than compute Meta AI 2024↓Introducing Meta Llama 3: The most capable openly available LLM to dateMeta AI, 2024. The Llama 3 paper gives the 8B model 32 layers and 8 key-value heads under grouped-query attention Grattafiori 2024↓The Llama 3 Herd of ModelsAaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, 2024. In fp16 that works out to a KV cache of about 128 KB per token, so memory grows linearly with context length.

The latent column is where planning gets cheap enough to run on a robot. V-JEPA 2-AC pairs a frozen 1B-parameter ViT-g video encoder with a 300M-parameter block-causal predictor, so the whole planning stack fits in a few GB Assran 2025↓V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and PlanningMahmoud Assran, Adrien Bardes, David Fan et al., 2025. Its planner runs the cross-entropy method with 800 samples over 10 refinement iterations and produces an action in about 16 seconds, while a Cosmos video-generation planner took about 4 minutes per action with only 80 samples. The paper reports the two times, not a ratio; they work out to roughly 15x Assran 2025↓V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and PlanningMahmoud Assran, Adrien Bardes, David Fan et al., 2025. Cheap per step does not mean flat in horizon. The predictor is block-causal: each patch feature attends to the patch features, actions and end-effector states of the current and earlier steps, so a longer rollout carries a longer context, though of 16 x 16 latent patch maps rather than pixels. Both planners in that comparison looked one step ahead Assran 2025↓V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and PlanningMahmoud Assran, Adrien Bardes, David Fan et al., 2025.

The generative-video column is the expensive one. Cosmos Predict spans autoregressive models at 4B to 13B and diffusion models at 7B to 14B over continuous and discrete video tokenizers NVIDIA 2025↓Cosmos World Foundation Model Platform for Physical AINVIDIA, 2025. NVIDIA's own measurements put Predict2's 14B Video2World variant at 56 GB of VRAM and roughly 85 seconds on a GB200 to over 30 minutes on a DGX Spark per clip at 480p and 16 fps, with 720p slower still NVIDIA 2025↓Cosmos-Predict2 Inference Performance and GPU Memory RequirementsNVIDIA, 2025; the Predict1 14B Video2World card lists more than 80 GB without offloading for a five-second 1280x704 clip NVIDIA 2025↓Cosmos-1.0-Diffusion-14B-Video2World Model CardNVIDIA, 2025. These are offline generators rather than 24 fps runtimes, and a policy that calls one per action step inherits that latency.

The explicit-3D column inverts the trade. 3D Gaussian Splatting stores the scene as rasterizable primitives, so a query is a raster pass: no weights, no tokens, and tens of milliseconds or better per frame on ordinary GPUs Kerbl 2023↓3D Gaussian Splatting for Real-Time Radiance Field RenderingBernhard Kerbl, Georgios Kopanas, Thomas Leimkühler et al., ACM Trans. Graph. 42(4), author manuscript (2023). Memory moves to the asset itself, typically a few hundred MB to a few GB per captured scene, and hundred-million-splat scenes outgrow consumer cards until level-of-detail or offload steps in. The limitation is semantic rather than computational: a splat answers what a view looks like, not what an action does, which is why the real-to-sim-to-real pattern below attaches explicit physics.

Two rows carry the design lesson. Batch behavior is the tell for robot use: LLM decode batches KV-bound streams, CEM planning is built to be batched, video diffusion usually runs one clip at a time, and rasterization makes extra cameras nearly free, which is why splats are attractive for evaluation and sensor simulation. The horizon row is a difference of degree: the latent predictor's context also grows with each step, by one 256-token patch map at a time, while token generators pay linearly in both KV and steps. For a training loop that asks the model for thousands of imagined rollouts, as in the model-based RL guide, those two rows decide what is affordable.

Where physics remains difficult to replace

Planning queries actions the dataset did not take. A physics engine can usually integrate those actions as long as its modeled regime remains valid. A learned model is being asked to extrapolate. A planner can exploit that extrapolation error by selecting actions whose predicted reward is high only because the model is wrong.

Safety constraints also favor explicit state. Joint limits, collision geometry, force bounds and signed distances can be checked directly. A video prediction may imply a collision without representing the quantity a safety monitor needs.

Throughput is workload-dependent. GPU physics can advance thousands of low-dimensional environments in parallel. A large video diffusion model may spend several denoising passes on one rollout. Learned latent dynamics can be faster than either. “Neural” does not determine speed; state size, horizon and sampling procedure do.

Where learning has the advantage

Rendering a real workcell with every texture, reflection, cable and background object is expensive. A learned visual model can absorb that distribution from video. Contacts with deformable objects, unmodeled actuator dynamics and wear can also resist clean parameterization.

Robotic World Model is an example of pushing the learned system toward a neural simulator for policy optimization Li 2025↓Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in RoboticsChenhao Li, Andreas Krause, Marco Hutter, 2025. RoboWorld pushes toward policy evaluation, using a neural simulator plus task-progress scoring and validating rankings against real robot results Jeon 2026↓RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy EvaluationByeongguk Jeon, Seonghyeon Ye, JaeHyeok Doo et al., 2026. Their usefulness depends on downstream agreement, not on replacing a physics engine in name.

Three hybrid patterns

Learn appearance, keep dynamics

Real-to-sim systems reconstruct a scene's visual geometry while retaining a conventional physics engine for motion and contact. RoboGSim uses Gaussian-splatting-based scene reconstruction in a real-to-sim-to-real pipeline Li 2024↓RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting SimulatorXinhai Li, Jialin Li, Ziheng Zhang et al., 2024. The learned renderer makes observations resemble the workcell; it does not by itself supply action-conditioned physics.

This pattern is strong for visual policy transfer when rigid-body mechanics are adequate. It is weak when the missing effect is friction, deformation or tool compliance.

Generate content, keep the engine

Generative simulation uses models to author objects, layouts, tasks and assets inside a physics simulator. RoboCasa supplies large-scale everyday manipulation environments Nasiriany 2024↓RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist RobotsSoroush Nasiriany, Abhiram Maddukuri, Lance Zhang et al., RSS 2024, and RoboCasa365 expands the framework for training and benchmarking generalist robots Nasiriany 2026↓RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist RobotsSoroush Nasiriany, Sepehr Nasiriany, Abhiram Maddukuri et al., ICLR 2026.

The generator solves a coverage problem while the engine still solves transitions. This avoids asking a video model to maintain contact causality over long rollouts.

Learn residual dynamics

Start from explicit dynamics and learn the difference between simulated and measured transitions. Residuals can target actuator delay, torque response, contact impulses or observation noise. This is the most conservative hybrid because the learned component has a named error to fit.

Residual models still need support checks. A correction learned around nominal walking may not extrapolate to a fall. The explicit base model provides a fallback, not a guarantee.

Evaluation must use the same real anchor

Both simulator types can be wrong in correlated ways. A policy may exploit contact softness in physics or visual shortcuts in a world model. Evaluate the environment by whether policies and interventions transfer.

SIMPLER formalizes this for conventional simulation: build matched environments and measure correlation between simulated and real policy performance Li 2024↓Evaluating Real-World Robot Manipulation Policies in SimulationXuanlin Li, Kyle Hsu, Jiayuan Gu et al., 2024. Apply the same protocol to learned worlds. Include several policy families, physical perturbations and held-out environments. The world-model evaluation guide details the ranking and calibration tests.

Do not compare a physics engine on task success with a world model on frame similarity. Put both behind the same decision: select a policy, select an action sequence or generate training experience, then measure the real consequence.

A selection rule

Use explicit physics when the state and mechanics can be specified, broad counterfactuals matter, constraints must be inspected, or massive parallel stepping is the main value. Use a learned world model when real observations contain structure the simulator cannot economically author, available trajectories cover the decisions of interest, and downstream validation can anchor model error.

Use a hybrid when the mismatch is localized. That is the default worth defending: geometry, constraints and reliable mechanics stay explicit; appearance, residual dynamics and task distributions are learned where data has an advantage. The result is not philosophically pure. It is easier to validate, which is the property robot learning needs.