World-model evaluation is the testing of a robotics world model against the decisions a robot must make, from one-step action responses to agreement with real policy rankings. Image metrics cannot establish that: on Physics-IQ, physical understanding in video generators was "unrelated to visual realism"Source 6.

Two panels: real robot evaluation, labelled expensive, slow and difficult to reproduce, with photographs of a robot arm at a table; and simulated evaluation, labelled cheap, scalable and fully reproducible, with matching simulated views.
A cheaper evaluator is useful only if it agrees with reality. SIMPLER builds simulated scenes matched to real ones and checks whether simulated policy performance correlates with real performance; a learned simulator should meet the same test.
Paper figure: Xuanlin Li et al. / arXiv. Licence: CC BY 4.0.

The central failure is action indifference: smooth, photorealistic futures that ignore the robot's action. A 2026 survey of robot world models rates controllability above visual plausibilitySource 1, and the closed-loop World-in-World benchmark found it mattered more than visual quality for task successSource 7.

Self-check

You want to use a video world model to select between two robot policies before hardware testing. Which validation result matters most?
Read the reasoning
  • Its policy ranking agrees with repeated real-world policy trialsThis directly validates the intended use. The neural evaluator may still be imperfect, but its ordering is calibrated against the decision it will support.
  • Its generated frames receive the highest perceptual sharpness scoreSharpness may improve human inspection, but it does not show that the model preserves action consequences or ranks policies the way the real robot does.
  • It can generate the longest uninterrupted videoDuration is useful only if errors remain bounded and actions stay causal. A long action-insensitive rollout can be less useful than a short calibrated one.

Validate the downstream decision: a policy evaluator must preserve real policy ordering. Attractive frames alone do not establish that.

Perception scores miss function, so each use needs its own test

WorldArena scores 14 models separately as data engines, policy evaluators and action planners, and reports a "significant perception-functionality gap"Source 8. NVIDIA reports its DreamGen Bench video scores tracking downstream policy successSource 9. 1X saw unchanged policy weights degrade within days after subtle lighting or background changesSource 10.

Level 1: open models often fail to follow a changed action

Build matched counterfactuals: fix the observation, noise, prompt and history, and change one action dimension. WorldEval found raw action inputs often yield videos that ignore the actionSource 12, and IRASim and Ctrl-World add frame-level action conditioningSource 13 Source 14. Across six open action-conditioned world models and more than 18,000 test instances, WorldSimProbe found systematic degradation in how actions are realizedSource 30.

Level 2: pooled video metrics hide physical errors

Roll out autoregressively, score task events such as grasp, slip and release, and report error per horizon. Fréchet Video Distance "increases only slightly with large temporal corruption"Source 15. WorldModelBench uses 67K human labels to catch violations such as objects changing sizeSource 16, and video models trained on simple physics failed out of distribution, copying the closest training exampleSource 17.

A grid of video frames of a robot arm pushing objects at steps 0, 8, 14, 21 and 29, with rows for ground truth and five rollout methods; red boxes mark misaligned and blurred frames, and the last row is marked aligned.
Autoregressive rollouts can drift in ways one frame hides. On BAIR robot pushing, the RoboWorld authors mark other forcing schemes misaligned or blurred within 29 steps, while their step forcing stays aligned with the ground truth.
Paper figure: Byeongguk Jeon et al. / arXiv. Licence: CC BY 4.0.

Level 3: the progress scorer needs its own test

Many evaluators pair a video generator with a vision-language scorer, as WorldGym does, so each half needs its own testSource 18. GVL fixes poor naive VLM value prediction by ordering shuffled frames, scoring more than 300 real tasksSource 19. Runway and DreamDojo used human graders instead, Runway collecting over 16,000 ratingsSource 20 Source 21.

Level 4: the best simulators order policies as real trials do

A policy evaluator must reproduce reality's ordering, measured as Pearson correlation and Mean Maximum Rank Violation (MMRV)Source 20. Real comparisons often rest on 10 to 50 trials, and adding trials after peeking risks p-hacking; the STEP sequential test cuts trials by up to 32%Source 22.

A diagram of RoboArena: a pool of generalist policies feeds a ring of photographed robot workstations, each running pairwise comparisons on its own task, which aggregate into a ranked score table.
RoboArena ranks policies from double-blind pairwise trials that many evaluators run on their own scenes and tasks. More than 600 real-robot episodes on the DROID platform ranked seven generalist policies.
Paper figure: Pranav Atreya et al. / arXiv. Licence: CC BY 4.0.

Google DeepMind reports its Veo simulator ranked eight Gemini Robotics checkpoints against more than 1,600 real evaluations at MMRV 0.03, though its absolute success predictions ran lowSource 23. WorldGym reached r = 0.78Source 18, and NVIDIA reports r = 0.995 for DreamDojoSource 21. Runway reports Pearson 0.95 and MMRV 0.033 over eight RoboArena policies, testing ranking onlySource 20. PolaRiS found that Ctrl-World "produces heavy hallucinations during object interaction", mis-ranking policies at MMRV 0.22Source 24.

Physics simulation sets the bar. In June 2026, Yuke Zhu and colleagues reported that SimFoundry, which rebuilds a real scene in simulation from video, predicted real results across 7 tasks and 5 policy architectures at mean Pearson 0.911 and MMRV 0.018Source 5. PolaRiS reports r = 0.9 against real trialsSource 24, and X2Real 0.84Source 25.

Level 5: planning and fine-tuning in a world model improved outcomes

IRASim planning raised Push-T IoU from 0.637 to 0.961Source 13, and fine-tuning on Ctrl-World's imagined successes raised policy success by 44.7%Source 14. Post-training on action-observation data helped more than a stronger pretrained video generatorSource 7.

SIMPLER's test, whether simulated and real policy performance correlate under matched visual and physical conditions, applies to learned simulators tooSource 2. RoboArena widens real evaluation beyond one laboratorySource 4.

Saturated benchmarks hide robustness gaps

Use a factorial matrix to expose shifts one "unseen" split could conceal. RoboLab-120 sorts 120 tasks into visual, procedural and relational axes and measures sensitivity to controlled perturbationsSource 26:

ShiftExampleFailure it exposes
Visuallighting, texture, distractorsrepresentation dependence
Geometricobject pose, camera pose, layoutspatial grounding
Physicalmass, friction, compliancedynamics fidelity
Semanticnew object or instructionpretrained knowledge transfer
Policyarchitecture or checkpoint familyevaluator overfitting
Embodimentarm, gripper or control conventionaction-space dependence

LIBERO-Plus used such perturbations to uncover gaps a saturated benchmark hidSource 3, and PolaRiS found nearly all tested policies scoring 90 to 95% on LIBERO, poorly correlated with real performanceSource 24.

Disjoint data and blind trials guard against leakage

Foundation models can see the benchmark in pretraining video, and scorers can share a backbone with the policy. Ctrl-World trains on DROIDSource 14, which spans 564 scenesSource 27, and Runway's RoboArena ranking put pi05_droid firstSource 20. X2Real enforces "strictly disjoint training-evaluation pipelines"Source 25, and TRI compared policies in blind, randomized trialsSource 28.

A credible report pins the benchmark version and the real anchor

A credible report states the use case, action representation, horizon and real anchor, and separates generator from scorer error. It names the benchmark version: Physics-IQ Verified refined 57.6% of samples and shifted rankings at Kendall's τ = 0.46Source 29. By 1X's arithmetic, with a 15% real gap between two policies, a 70%-accurate world model picks the better one 90% of the timeSource 11.