Generative Simulation for Robot Learning

The previous three modules covered learned models that predict the future: in a latent space, in pixels, or in an embedding. This module is about a different division of labor. Instead of asking a generative model to be the simulator, ask it to author the simulator: the tasks, the scenes, the assets, the reward functions. The dynamics are left where they have always been, in a physics engine. Recall the two boundary verdicts from the taxonomy: a physics engine is a world model, just not a learned one, and a 3D Gaussian Splatting twin is not a world model, because it is a learned renderer bolted onto an engine's dynamics. Generative simulation takes that observation seriously. It keeps the engine and generates everything around it.

The reason that division of labor holds is below, in one scene with three layers. Turn off the physics proxy and push the mug: nothing happens, and nothing can, because a renderer has no dynamics to integrate. Turn it back on and the same push slides the mug by a distance the solver computes.

Push force4.0 N

mug displacement = 0.0 cm after 0 effective pushes

Appearance is on and simulation is on, physics proxy is off, so a 4.0 N push leaves the mug at 0.0 cm of displacement; the pixels have no mass until the physics proxy supplies a collision hull.

Current push-test layers

appearance: on
physics proxy: off
simulation: on
force: 4.0 N
displacement: 0.0 cm

Nothing moves. The appearance layer renders the mug but has no answer to a push: no collision hull, no friction, no integrator. A renderer is not a simulator. This is the failure mode of appearance-only digital twins and of generative dynamics that looks right but is not causally tied to the action.

120

kitchen scenes

RoboCasa, RSS 2024

100

tasks

systematic evaluation suite

2,500+

3D objects

150+ categories

2,500

RoboCasa365

kitchens, ICLR 2026

The 120-scene, 100-task, and 2,500-plus-object figures above describe RoboCasa v1, the RSS 2024 paper, not RoboCasa365 Nasiriany 2024 .

The argument: generate content, not dynamics

The generative video paradigm asks a model to learn physics from pixels. Its characteristic failures are physical-consistency failures: futures that look plausible but are not causally tied to the pending action. The 2026 world-model survey names weak action conditioning as the top open challenge in that line and warns that visual plausibility is only a weak proxy for control utility Hou 2026 . When the model's job is to produce the next frame, every error it makes is an error in the physics the policy will train on.

Generative simulation moves the generative model off that critical path. The model produces content: a task specification, a scene layout, a mesh, a reward function. A real solver then executes that content. Physical consistency is no longer something the model has to learn; it is something the solver enforces on every step, by construction. A generated mug might have the wrong friction coefficient, but it will never fall upward, pass through the table, or teleport, because the engine does not allow it. The failure modes shrink from "the physics is wrong" to "the content is off-distribution", and the second is a much more tractable problem.

This is why the approach is, today, the more reliable use of generative models in robot learning. The three systems below show the pattern at three scales: task generation, environment generation, and a full benchmark built from generated assets.

RoboGen: propose, generate, learn

RoboGen uses a propose-generate-learn cycle to create tasks and learn skills in simulation Wang 2024 . GPT-4 proposes tasks from a robot type plus a sampled object or example tasks. It generates scene configurations, including asset sizes, poses, and articulation states. Assets are retrieved from object databases; soft-body target shapes can instead be generated through an image-to-mesh pipeline. GPT-4 decomposes each task into sub-tasks, chooses a learning method, and supplies reward functions or target-shape supervision.

The available methods include reinforcement learning, motion-planned action primitives, and gradient-based trajectory optimization. The paper's main text presents RL as suitable for locomotion and contact-rich interaction, but Appendix A.3 says the reported locomotion implementation uses cross-entropy-method planning with the ground-truth simulator. Manipulation uses SAC and motion-planned grasp/approach primitives; soft-body optimization uses Adam. The paper's claim of minimal human supervision still leaves prompt design and in-context examples, and its evaluations include human inspection of scenes, rewards, and learned skills. The authors report scene and reward failures and identify large-scale skill verification and sim-to-real transfer as remaining limitations Wang 2024 .

Holodeck: language-guided environment generation

Holodeck (CVPR 2024) generates AI2-THOR environments from text prompts using GPT-4 Yang 2024 . It selects objects from Objaverse and proposes spatial-relational constraints such as a coffee table in front of a sofa. In the reported experiments, a depth-first-search layout solver enforces collision and room-boundary constraints while trying to satisfy as many soft relational constraints as possible. The appendix also describes a mixed-integer linear programming alternative. The example prompts include "office of a professor who is a fan of Star Wars" Yang 2024 .

The evaluation separates human judgments of asset selection and layout from an automatic CLIP-based visual-coherence score. For object navigation, a model pretrained on ProcTHOR-10K is fine-tuned in generated scenes and evaluated in artist-designed NoveltyTHOR environments. This is a simulated navigation experiment, not a test of physically valid manipulation. The system uses designed prompts and a curated asset library; the authors report difficulties with complex layouts and assets missing from Objaverse Yang 2024 .

RoboCasa: a benchmark built from generated assets

RoboCasa v1 reports 120 kitchen scenes built from architectural references: 10 floor plans combined with 12 styles. Its object library contains over 2,500 assets across more than 150 categories; Section III-C and Table I give the exact counts as 2,509 objects and 153 categories. Its 100-task suite comprises 25 atomic tasks and 75 composite tasks. The composite-task blueprints come from GPT-4 and Gemini 1.5, with human filtering, modification, and code implementation Nasiriany 2024 .

The object assets come from Objaverse and the text-to-3D service Luma.ai; scene textures are generated with MidJourney. For training data, RoboCasa adapts human demonstrations with MimicGen, retaining successful generated trajectories Nasiriany 2024 .

The scaling study is a specific simulation experiment, not a result over the entire 100-task suite. It trains a language-conditioned BC-Transformer for a Franka Panda on an Omron mobile base, using 100, 300, or 3,000 generated demonstrations per task across 24 atomic manipulation tasks; navigation is excluded. The experiment uses Objaverse objects and AI-generated training textures. Evaluation uses human-curated textures, unseen object instances, and 50 trials per task across five fixed kitchen scenes, two with unseen styles. Figure 12 reports overall success rates of 26.3%, 35.0%, and 47.6% as the generated-data size increases. Individual tasks do not improve monotonically. Section VIII-C refers broadly to datasets over 25 tasks, while Section V-A and Figures 6 and 12 explicitly define this generated-data comparison as 24 tasks Nasiriany 2024 .

RoboCasa365 extends the same platform to 365 everyday tasks across 2,500 kitchen environments, with 600+ hours of human demonstrations and 1,600+ hours of synthetically generated demonstrations, and with explicit support for multi-task learning, foundation-model training, and lifelong-learning benchmarks Nasiriany 2026 .

Isaac Lab v1 describes a related demonstration-generation workflow. Mimic segments human demonstrations into object-centric subtasks, rigidly transforms the segments, and recombines them for new poses. Once an environment implements the required pose/action, end-effector, and gripper interfaces, the paper says generation can start from as little as one human demonstration and run across parallel environments. This is not a promise of unlimited successful demonstrations: its SkillGen integration exposes trial counts, environment counts, and planner settings that trade speed against success rate and motion quality NVIDIA 2025 .

GRS starts from a single real-world RGB-D observation. SAM2 segments the image; depth and a calibrated robot-frame transform provide object bounding boxes, and a vision-language model matches the objects to supplied simulation-ready assets. It generates task code and tests, then uses a router to revise either until an oracle policy can complete the simulated task Zook 2024 .

This is a bounded real-to-sim authoring procedure, not a guarantee that every generated task is solvable: the experiments cap repair attempts, exclude runtime-error cases from the reported reward average, and document failures that can bypass intended behavior. Sim-to-real training and transfer remain future work in the inspected version Zook 2024 .

The push test: appearance is not physics

The figure at the top of this module is the whole argument in three layers. The appearance layer renders the mug and the table; it stands in for anything that only knows what the scene looks like, a neural reconstruction, a generated texture, a photorealistic video frame. The physics-proxy layer adds what a solver needs: collision geometry, a mass, a friction coefficient. The simulation layer shows the integrated result, sliding the mug by $d = v^2/(2\mu g)$ , every time, deterministically.

This is the mechanical version of the taxonomy's verdict on 3DGS twins. A learned renderer gives you layer one. Generative video tries to learn layer three directly, and its errors land in the physics. Generative simulation generates layer one and borrows layers two and three from an engine that already has them.

Limitations, stated plainly

Generated scenes are only as physically correct as their asset parameters. A text-to-3D mug looks like a mug; its mass, friction, and center of mass are estimates. Tuning simulated contact against real-world measurement remains manual work, which is why physically accurate simulation-ready assets are a commercial category rather than a solved problem.
Task fitness need not capture human intent. Eureka explicitly notes that its task metric may not align with the behavior a person wants. Its separate human-feedback experiment uses textual corrections to steer a simulated humanoid's gait. The authors also describe a generated PushBlock reward as "amenable to post-hoc human inspection and editing"; this is an interpretability claim, not a requirement that every reward be manually inspected or a comparative study of reward hacking Ma 2024 . The reward-design module explains the distinction.
Scene diversity has specific design sources. Holodeck's authors document cultural biases from the LLM and the asset-retrieval component, with only qualitative evidence that changing prompts can help Yang 2024 . RoboCasa's authors say they consulted home-design and architecture magazines to compile kitchen floor plans and styles Nasiriany 2024 .

Where this leaves the field

None of this makes the generative world-model paradigms wrong. It makes them premature as simulators of record. The evidence summarized in the generative video module shows learned models earning narrow roles in evaluation and data generation, while every one of their physical-consistency failure modes is a problem that a real solver simply does not have. The pragmatic reading for 2026: use generative models where their failure is cheap, authoring tasks, scenes, assets, and supervision, and use engines where their failure is expensive, integrating what happens when a robot acts. RoboGen, Holodeck, and RoboCasa are what that division of labor looks like when it ships Wang 2024 Yang 2024 Nasiriany 2024 .

Draw the hybrid boundary explicitly

A productive generative-simulation stack assigns each layer to the component with testable strengths. A generator can propose layouts, tasks, textures, objects and initial states. An asset pipeline can reject invalid geometry and attach measured or bounded physical properties. A physics engine can integrate contacts and constraints. A scripted or learned supervisor can label outcomes. The final dataset should record which component produced every field.

Validate at the boundary, not only at the end. Check asset scale and collision geometry before dynamics, static stability before policy rollouts, and contact parameters against a small set of real measurements before claiming transfer. Keep generated scenes that fail as adversarial tests rather than silently filtering them from reports. The world-model and simulator comparison provides the selection rule, while the evaluation guide separates visual plausibility, predictive accuracy and downstream policy evidence.

Generative Simulation