A technical primer on world models

Large language models (LLMs) can describe a glass falling off a table in vivid detail, but they don't understand gravity, friction or momentum. Language is a lossy, one-dimensional encoding of a multimodal reality. World models aim to close that gap by learning what it means to make something happen, not just describe it.

What are world models?

Simplistically, similar to how a language model is trying to predict the next word and learns a representation of the language, a world model tries to predict what's going to happen next in the world based on the sequence of actions that an agent is performing. It’s simulating an entire environment, moment by moment, in reaction to an agent. Through this, the model learns a representation of the world.

Formally, a world model is an interactive predictive system that simulates spatial-temporal environments. It learns to predict the next state s' given the current state s and a specific action a: P(s' | s, a). This action-conditioned prediction enables an AI system to:

• Perceive: “What’s happening now?”
• Predict: "If I do X, what happens?"
• Plan: "What sequence of actions achieves goal G?"
• Reason: "Why did that happen? What caused it?"
• Act: "What action should I take next?"

Where language models learn correlations that ‘cup’ co-occurs with ‘table’ and ‘spill’, world models aim to learn causality. For example, pushing a cup moves it and moving it past the table's edge makes it fall. Although there are lots of historical papers on world models, one that popularized the term in the developer community is Ha and Schmidhuber's 2018 paper, World Models.

What practitioners mean by world model

World model is an overloaded term in AI research, referring to different concepts across subfields. While definitions vary, they generally involve AI systems learning predictive representations of environments for simulation, planning and action. The major lineages include:

• Reinforcement learning origins: Richard Sutton's Dyna algorithm (1990) combined model-based RL with online learning. Modern descendants like DeepMind's DreamerV3 use recurrent networks to predict latent states, rewards and outcomes from observations and actions.
• Gaming and interactive environments: Generative systems such as DeepMind's Genie 3 train on unlabeled video and produce controllable virtual worlds from text prompts or images, with real-time interaction. The agent intervenes mid-rollout and the model updates accordingly.
• Video generation models: Sora, Veo and Runway Gen-4.5 synthesize realistic sequences from text prompts, implying learned dynamics. Whether they qualify as world models is actively debated. Proponents argue that high-fidelity video implies learned physics. Critics argue these models typically lack explicit action-conditioning, the ability to respond to interventions mid-sequence.
• JEPA and latent prediction: The central bet is that predicting every pixel of the future is intractable in any stochastic environment. JEPA sidesteps this by predicting in a learned latent space instead. V-JEPA trains a context encoder that maps spatiotemporal video patches to representations and a predictor that forecasts the embeddings of masked regions. A generative model that reconstructs pixels is forced to commit to low-level details (exact texture, lighting, leaf position) that are inherently unpredictable. By operating on abstract embeddings, JEPA can capture "the ball will fall off the table" without having to hallucinate every frame of it falling.
• Robotics and physical AI: Embodied systems often use world models as differentiable simulators for internal rollouts before real-world action. For instance, Meta's Navigation World Model takes the problem of navigation, predicting future visual observations from past observations and navigation actions, then planning trajectories by simulating them.

Across these lineages, the shared idea is to compress state into a latent representation and model its dynamics.

The gap between generating video and simulating a world

Video generation models are increasingly treated as proto-world models. This is for good reason since they learn dynamics, motion and spatial relationships from data at scale. But simulating an environment that an agent can act within requires capabilities that most video models do not yet fully possess. Five properties mark the gap, including causality, interactive controllability, persistence, real-time responsiveness and physics generalization. Each maps to a specific data problem and failure modes.

1. Causality: Most SOTA video models use bidirectional attention. They process whole sequences at once, so future frames can influence past ones. That works for offline rendering and breaks the moment a user or agent has to intervene mid-rollout. Causal training requires per-timestep independent noise levels and sequential frame-by-frame supervision. The data problem here is annotation: hierarchical temporal captioning with timestamped event descriptions ("0–5s: camera pans left, 5–10s: object enters frame"), discrete interventions paired with their immediate consequences and a curriculum organized by sequence length to teach longer-horizon reasoning.
2. Interactive controllability: A causal model that does not respond to actions is a video player. The shape of "action" is application-specific, ranging from keyboard inputs for a game like Genie 3, motor commands for a robot, steering and throttle for an AV. The bottleneck is action-aligned video. Most internet video is action-free. Latent action models such as LAPA and LAWM learn unified action representations from unlabeled video, then align them to a small labeled trajectory set. That changes the cost structure, as action-free contributor footage becomes a useful training signal. Dense temporal captioning of subtask intervals (Joel Jang's "reaching for the cup, grasping the handle, pouring") matters more than dense bounding-box labeling.
3. Persistence: Real applications need extended or indefinite-length sequences that stay internally consistent. Longer context windows alone fail under interactive latency. RoboWM-Bench documents the embodied execution failures: spatial reasoning errors, unstable contact prediction and non-physical deformations when generated behaviors are run on real robots. Training data fixes for persistence include frame packing, needle-in-a-haystack memory pairs where early-frame details must influence late predictions, multi-view video with explicit scene geometry and clean static-versus-dynamic labeling.
4. Real-time responsiveness: Tolerable latency varies by application, from roughly one second for live streaming, 100 ms for gaming and 10 to 20 ms motion-to-photon for VR. A non-causal model that generates a t-second clip in one shot has a minimum latency of t seconds regardless of throughput, which is why frame-by-frame generation is non-negotiable for interactive systems. Training data has to mirror inference conditions, including autoregressive generation with KV caching, few-step distillation pairs and hardware-aware curation across complexity levels.
5. Physics generalization: Current models extrapolate poorly to out-of-distribution scenarios (objects falling at atypical speeds, grippers at unfamiliar pressures). Visual plausibility is not physical correctness. Data fixes span physics-annotated sets (mass, velocity, friction, material) with ground truth from MuJoCo or Isaac Sim, paired sim and real examples plus residual learning data and diverse domains across robotics, AVs, human motion and natural phenomena.

These five properties describe the gap as seen from the video generation side, but video is not the only angle of attack. Robotics and physical AI teams are approaching the same destination from embodied experience: collecting massive volumes of egocentric data from real and simulated environments, learning contact dynamics through touch and force feedback, and grounding representations in proprioceptive signals that video alone cannot capture. Latent prediction approaches like V-JEPA sidestep pixel generation entirely, betting that abstract representations will scale further than reconstruction-based methods.

As the world model field converges, it will likely draw from all of these lineages.

How world models transform AI agent training

World models change agent training by turning a learned simulator into the primary venue for experience. Instead of collecting every lesson through real-world trial and error, an agent can rehearse inside its own predictive model, compressing the cost, risk and time of learning. The term "agent" here means an embodied or simulated system that takes physical actions in an environment (a robot arm, an autonomous vehicle, a game-playing policy), not just the LLM-based digital agents now common in software workflows.

Sample efficiency and data reuse

One of the biggest transformations world models bring is improved sample efficiency, where the agent can reuse logged data many times to improve its model and train policies without needing fresh interactions for every policy update. Because the world model is a differentiable generative model, it can generate arbitrarily many imagined trajectories at low marginal cost, turning a fixed dataset into a virtually unbounded training resource for decision‑making. This is particularly valuable in domains where real interaction is expensive, slow or risky, such as robotics, autonomous driving and network control.

Safer and cheaper training via internal simulation

World models allow agents to “mentally” simulate many candidate futures before committing to an action in the real world, which reduces the risk of catastrophic failures during learning. Instead of trying a potentially dangerous maneuver on a real robot or vehicle, the agent can evaluate it inside the world model and discard it if predicted outcomes are bad. This internal simulation paradigm greatly cuts the cost and risk of training, both in physical systems (where hardware can be damaged) and in digital environments with high user or compute costs.

Enabling long‑horizon planning and reasoning

By learning latent dynamics across many steps, world models let a planner or actor-critic search for action sequences that maximize value. DreamerV3 demonstrates this at scale, outperforming specialized methods across more than 150 diverse tasks with a single fixed configuration.

Better generalization and out‑of‑distribution robustness

World models also improve generalization, especially when combined with representation learning techniques that enforce invariance to irrelevant variations. Methods that use contrastive learning and auxiliary tasks (such as depth prediction) train the world model to focus on causal, task‑relevant features that remain stable across changes in appearance or style.

These benefits are strongest in domains where visual dynamics are smooth and contact physics is simple. In contact-rich manipulation, high-frequency tactile feedback and scenarios with sharp discontinuities, learned world models remain unreliable and may offer limited advantage over model-free approaches.

What is still open?

World models are a complementary path to general-purpose AI capability. The five properties above describe where world models fall short today. Closing those gaps is also a data problem. The training data the field still lacks falls into three categories, in no particular order:

• First, 3D reconstruction and environment creation pipelines that turn real-world and synthetic scenes into structured, reusable training assets.
• Second, AI-ready game worlds that accelerate environment authoring and give agents cheap, diverse training substrates.
• Third, world model training data itself: causal and counterfactual reasoning sets, vision-language-action (VLA) data that links perception to instruction and motor outcome, diverse environment variations per scene for robust behavior under distribution shift, and chain-of-thought annotations for spatial, temporal and game logic.

Talk to our team to build the data foundations of your physics-aware interactive AGI future.