# World Models: Can AI Simulate Reality? > What a world model is, the evidence that AI builds one inside, the LeCun debate, and what Sora and Genie can and can't simulate yet. - Author: Tai Pham Phu - Published: 20 Oct 2026 - Tags: How AI Works, LLM, Applied AI, World Models, Video Generation - URL: https://phamphutai.com/2026/10/20/world-models/ In February 2024, OpenAI showed the world Sora, a model that turned a sentence of text into a minute of realistic video. The clips went everywhere: a woman walking down a rainy Tokyo street, woolly mammoths in the snow. The technical report that came with it had an ambitious title: **"Video generation models as world simulators."** It listed abilities the team said had emerged just from training at scale. The camera could move and the scene stayed consistent in 3D. People and objects could leave the frame and come back. Sora could simulate simple actions that change the world. One example: **a man eats a burger and leaves bite marks.** It could even imitate Minecraft. Then, in its closing discussion, came the honest part. The same report said Sora **does not correctly model the physics of many basic interactions, like glass shattering.** Other actions, *like eating food*, don't always produce the correct change in the object. Long videos develop inconsistencies, and objects sometimes appear out of nowhere. OpenAI's own announcement page gave a similar example: a person bites a cookie, and afterwards the cookie may have no bite mark. Even so, the report closed on optimism: scaling video models, it said, is a promising path toward general-purpose simulators of the physical world. So in one document, the burger sometimes gets bite marks and sometimes doesn't. That's the whole question of this post in miniature. **Is the model simulating the world, or just painting what a world usually looks like?** It's the last part of this series, and it's one of the biggest open questions in AI. --- ## What "world model" means A world model is an internal model an AI uses to **predict what happens next**, usually given what it just did. You have one. Before you push a glass toward the edge of a table, you already know roughly what happens if it goes over. You didn't need to try it. You ran a tiny simulation in your head. Planning, imagining, and noticing "that's weird" all rely on it. In AI research, the term has a fairly specific history. ### Ha and Schmidhuber, 2018 In March 2018, David Ha and Jürgen Schmidhuber published a paper simply called **"World Models."** Their agent had three parts: 1. **V (vision):** a small network that compresses each video frame into a short list of numbers. A compact summary of "what I see." 2. **M (memory):** a recurrent network that predicts what that summary will look like in the next frame, given the action the agent takes. This is the world model. 3. **C (controller):** a tiny decision-maker that picks the next action, using V's summary and M's memory. The striking result was in a simple version of the game Doom, where the agent must dodge fireballs. They trained the controller **entirely inside the world model's imagination**, a "dream" version of the game generated by M. Then they put it in the real game, and the skill transferred. It survived well past the threshold that counts as solving the task.

The classic world-model loop: compress what you see, predict what your actions will do, rehearse in imagination, then act.

### Model-based reinforcement learning That idea belongs to a family called **model-based reinforcement learning.** Reinforcement learning is learning by trial and reward. "Model-based" means the agent also learns a model of how its environment works, so it can practise in imagination instead of only by real trial and error. Real trial and error is slow, and for robots or cars, expensive and dangerous. The most impressive recent example is **DreamerV3**, by Danijar Hafner and colleagues, published in *Nature* in April 2025. One algorithm, with one fixed set of settings, beat specialised methods across more than 150 tasks. It was, to the authors' knowledge, the first algorithm to collect diamonds in Minecraft **from scratch, without human data**: no gameplay videos, no step-by-step help. Its trick is exactly the one above. It learns a model of the world and improves its behaviour by imagining future scenarios. **An analogy, and where it breaks.** A world model is like a flight simulator the AI builds for itself, then trains in. The analogy breaks in an important way: a real flight simulator is written by engineers from the laws of physics. A learned world model is only as good as the data it saw. Ask it about situations it never saw, and it may confidently simulate nonsense. A pilot trusts the simulator because someone checked it. Nobody has checked the AI's. --- ## Do language models have world models inside? This is where the debate gets interesting. Language models were never told to build a world model. They were only trained to predict the next token. But maybe, to predict text about the world well, you have to model the world a bit. ### Othello-GPT The cleanest test is a board game. In a paper presented at ICLR 2023, Kenneth Li and colleagues (including David Bau and Martin Wattenberg) trained a GPT-style model on sequences of legal moves in **Othello**, the black-and-white disc game. It never saw the board. It was never told the rules. Just move lists, like "E3, F4, ...", and the task: predict a legal next move. Then they looked inside (Part 1 of this series explains how). They found that the model's internal activations encoded **the current state of the board**, which squares were black, white or empty. And it wasn't a side effect. When they edited that internal board, changing a square from one colour to another inside the model's activations, **the model's predicted moves changed to match the edited board.** That's evidence the representation was actually being used. A follow-up in September 2023 by Neel Nanda, Andrew Lee and Martin Wattenberg made it even cleaner. The original paper needed a complex probe to read the board. Nanda's team found the model thinks in terms of **"my pieces" versus "the opponent's pieces"**, not black and white. Read that way, the board was stored in a simple, linear form, and they could steer the model with simple arithmetic on those directions. ### Space and time in Llama In October 2023, Wes Gurnee and Max Tegmark looked at Meta's Llama-2 models. Given names of places (cities, landmarks, places in New York) and events (historical figures, artworks, news headlines), the models' internal activations contained **simple, linear representations of latitude, longitude and dates.** They even found individual neurons that track spatial and temporal coordinates. Their careful conclusion: models have basic ingredients of a world model, and more work is needed. ### But: a map that isn't a map Now the important counterweight. In 2024, Keyon Vafa, Jon Kleinberg, Sendhil Mullainathan and colleagues trained transformers on turn-by-turn directions from real New York City taxi rides in Manhattan. The models were excellent at the task. Their predicted next turns were nearly always legal, and they produced valid routes 96 to 99% of the time. Then the researchers reconstructed the street map the model seemed to be using. **It wasn't Manhattan.** It had streets in physically impossible directions and roads passing over other roads. And when they added detours, the kind any driver handles every day, performance collapsed. For the model trained on shortest paths, a 1% chance of a random detour dropped valid routes to 69%. At 10%, almost nothing worked. Their conclusion is the most useful sentence in this whole debate: **a model can do a task very well without a coherent world model, and that incoherence makes it fragile when things change.** So the honest answer to "do LLMs have world models?" is: **pieces of them, sometimes, of uneven quality.** Real, usable internal structure exists, as Othello-GPT showed. It's also patchy, and good performance on the usual tests doesn't prove it's coherent. --- ## The debate: pixels, tokens, and Yann LeCun The strongest critic of "just scale the generator" is Yann LeCun, Turing Award winner, who spent twelve years at Meta, most recently as its chief AI scientist, before leaving in late 2025. His position, stated as fairly as I can: - **Generating every detail is the wrong goal.** In his 2022 position paper, *A Path Towards Autonomous Machine Intelligence*, he argued that a world model shouldn't try to predict every pixel. The world is full of details that are unpredictable and don't matter, like exactly how each leaf on a tree will move. A model forced to predict them wastes effort and gets blurry or wrong. When Sora launched, he wrote on X that modeling the world for action by generating pixels is "as wasteful and doomed to failure" as an older, largely abandoned approach in computer vision. - **Predict in an abstract space instead.** His proposed design is called **JEPA**, Joint Embedding Predictive Architecture. Instead of predicting the next frame or the next word, it predicts the *summary* (the internal representation) of what comes next. It's allowed to ignore what doesn't matter. - **Autoregressive LLMs accumulate errors.** In talks since 2023, he has argued that a model generating one token at a time, with no way to plan or check its whole answer, has a small chance of going wrong at every step, and those chances add up over a long answer. Language, he argues, is also a thin slice of what intelligence needs. A child learns a huge amount about physics by watching, long before reading.

Two bets on world models: predict every pixel, or predict a compact summary and let the unpredictable details go.

He has backed the argument with work. In June 2025, Meta released **V-JEPA 2**. It learned from over a million hours of internet video, then got a small extra training step on under 62 hours of unlabelled robot video. That was enough to let robot arms in two different labs **pick up and place objects zero-shot**: in labs it had never seen data from, with no task-specific training or reward. In late 2025, LeCun announced he was leaving Meta to start a company built around world models. It's called AMI Labs, based in Paris, and it was widely reported in 2026 to have raised around a billion dollars. **The other side, also fairly.** The generative camp's answer is that predicting pixels or tokens forces a model to learn whatever structure it needs, and scale keeps improving it. The Othello and space-and-time results show next-token training can produce real internal models. Critics of the error-compounding argument point out that modern models routinely catch and fix their own mistakes mid-answer, especially reasoning models. And the generators keep getting better. When OpenAI launched **Sora 2** at the end of September 2025, it said the model handled physics noticeably better: a missed basketball shot now bounces off the rim instead of teleporting into the hoop. My read: both camps agree on the destination. AI needs good models of how the world works. They disagree on the route. Nobody has proven the other side wrong yet. --- ## Interactive world models: you can walk around in them Video generators make clips you watch. The newer frontier is models you can **act inside**. Press a key, and the model generates what happens next, frame by frame, in response. Google DeepMind's **Genie** line is the clearest public example: - **Genie (February 2024).** An 11-billion-parameter model trained on about 30,000 hours of 2D platform-game videos from the internet, filtered from a larger pool. The clever part: those videos had no record of which buttons were pressed. Genie learned its own set of "latent actions," basically guessing the controls from watching. Give it one image, even a sketch, and you get a playable 2D world. - **Genie 2 (December 2024).** Playable 3D worlds from a single image, controlled with keyboard and mouse. It modelled things like opening doors, water, smoke and gravity, and remembered parts of the world that went out of view. Worlds stayed consistent for up to a minute, though most examples were 10 to 20 seconds. DeepMind used it to test agents like its SIMA game agent. - **Genie 3 (August 2025).** Worlds generated from text, explorable **in real time at 24 frames per second in 720p**, staying consistent for a few minutes. It remembers what it showed you up to about a minute back, so if you walk away and come back, the wall you painted is still painted. You can also add "world events" mid-session, like changing the weather. - **Project Genie (January 2026).** A prototype in Google Labs, built on Genie 3, opened to Google AI Ultra subscribers in the US aged 18 and over. Each generation is capped at 60 seconds. DeepMind is unusually clear about limits. For Genie 3 it listed: a narrow range of actions you can take, trouble modelling several independent agents interacting, no accurate simulation of real places, text that's only readable if you put it in the prompt, and interaction measured in minutes, not hours. For Project Genie it added that worlds may not follow real physics or match prompts, and controls can lag. A Google spokesperson also told reporters it isn't a game engine. The stated goal is not mainly games. It's **training grounds for AI agents and robots**: endless, varied practice worlds that would be impossible to build by hand. That's the Ha and Schmidhuber dream-training idea, at a much bigger scale. --- ## What's real and what's hype Let me sort it out plainly. **Real:** - Learned world models work well in **narrow, well-defined environments.** DreamerV3 mining Minecraft diamonds from scratch is a genuine result. - Language models build **real, usable internal structure** about their inputs, like Othello boards and coordinates. This is measured, not imagined. - Interactive video worlds now run **in real time for minutes**, which was science fiction a few years ago. - Video-trained models are starting to help **real robots** with simple tasks, like V-JEPA 2's pick-and-place. **Hype, or at least not yet:** - "Sora understands physics." Its own report said otherwise. Looking right is not the same as being right. The bite mark that sometimes appears and sometimes doesn't is the tell. - "LLMs have a model of the world" as a blanket statement. The taxi-map study shows good task performance can hide an incoherent model. - "World models will replace physics engines and simulators soon." Today's are consistent for minutes, struggle with multiple agents, and can't faithfully simulate real places. A physics engine is boring, exact and checkable. That still matters. - Any claim that one approach, generative or JEPA, has won. It's an open race. --- ## What it means in practice If you **build** with AI: - **Don't assume the model "knows how things work."** A model can describe a process perfectly and still fail when one step changes, just like the taxi model with detours. Test with variations and surprises, not just textbook cases. - **Keep real rules where you have them.** If your business has an exact rule (a booking can't overlap another booking, stock can't go negative), enforce it in code. Let the AI handle the fuzzy parts. A learned model is a good guesser, not a source of truth. - **For robotics and simulation, watch this space closely**, but plan pilots, not production. The progress from 2018 to now has been real and fast. If you **buy** AI: - When a demo says the AI "simulates" or "understands" the world, ask what happens in the cases it wasn't shown. The difference between a beautiful clip and a reliable system is the same as in any AI demo. - Ask about duration and consistency. "It stays consistent for a few minutes" is honest and useful. "It understands physics" is marketing. --- ## Key ideas - A **world model** is an AI's internal model for predicting what happens next, especially in response to its own actions. It lets an agent rehearse in imagination before acting. - The idea is old and proven in narrow settings: Ha and Schmidhuber trained an agent in its own "dream" in 2018, and DreamerV3 mined Minecraft diamonds from scratch. - Language models do build **real internal models** (Othello boards, space and time), but they can be **patchy and incoherent**, as the NYC taxi-map study showed. - **LeCun's critique:** predicting every pixel or token is wasteful and errors compound. His alternative, JEPA, predicts compact summaries instead. The generative camp bets that scale fixes it. Neither side has won. - **Interactive world models** like Genie 3 run in real time for minutes, with clear stated limits. Their main purpose is training agents and robots. - Looking realistic is not the same as simulating reality. **Test for what happens when things change.** --- ## That's the series: what we covered This was the last of twelve parts in *How AI Works: Advanced*. If you read them all, thank you. Here's the full map: 1. [Interpretability: How Scientists Look Inside an AI's Mind](https://phamphutai.com/2026/10/20/interpretability-looking-inside-an-ai/) 2. [Mixture of Experts: Why Giant Models Only Use Part of Their Brain](https://phamphutai.com/2026/10/20/mixture-of-experts-explained/) 3. [Long Context and the KV Cache: How AI Reads a Whole Book at Once](https://phamphutai.com/2026/10/20/long-context-and-the-kv-cache/) 4. [Shrinking AI: Quantization and Distillation Explained](https://phamphutai.com/2026/10/20/quantization-and-distillation/) 5. [Why AI Runs on GPUs](https://phamphutai.com/2026/10/20/why-ai-runs-on-gpus/) 6. [How AI Sees: Vision Transformers and CLIP](https://phamphutai.com/2026/10/20/how-ai-sees-vision-transformers-and-clip/) 7. [How Image Generators Work: Diffusion Explained](https://phamphutai.com/2026/10/20/how-image-generators-work-diffusion/) 8. [Glitch Tokens: The Strange Words That Broke GPT-3](https://phamphutai.com/2026/10/20/glitch-tokens-solidgoldmagikarp/) 9. [Reward Hacking: When AI Wins the Wrong Game](https://phamphutai.com/2026/10/20/reward-hacking/) 10. [AI Benchmarks and Goodhart's Law: Why Leaderboards Mislead](https://phamphutai.com/2026/10/20/benchmarks-and-goodharts-law/) 11. [Synthetic Data and Model Collapse](https://phamphutai.com/2026/10/20/synthetic-data-and-model-collapse/) 12. World Models: Can AI Simulate Reality? (you're here) If there's one thread running through all twelve, it's this: **these systems are not magic, and they're not just autocomplete either.** They're strange, engineered things that we can study, measure and understand, piece by piece. Every part of this series started with a real paper or a real event, and almost every one ended with an open question. That's not a weakness. It means the field is still young, and the people who take the time to understand how it really works, like you just did, are the ones who'll build the best things with it. Go build something. Test it on the hard cases. And keep asking how it works. *Sources: [OpenAI, "Video generation models as world simulators", February 2024](https://openai.com/index/video-generation-models-as-world-simulators/) (limitations as summarised by [The Decoder](https://the-decoder.com/openais-sora-is-much-more-than-a-text-to-video-generator/) and [Futurism](https://futurism.com/openai-sora-ai-simulate-worlds)); [Ha and Schmidhuber, "World Models", arXiv, March 2018](https://arxiv.org/abs/1803.10122) and [interactive version](https://worldmodels.github.io/); [Hafner et al., "Mastering diverse control tasks through world models", Nature, April 2025](https://www.nature.com/articles/s41586-025-08744-2); [Li et al., "Emergent World Representations", ICLR 2023](https://arxiv.org/abs/2210.13382); [Nanda, Lee and Wattenberg, "Emergent Linear Representations in World Models", September 2023](https://arxiv.org/abs/2309.00941); [Gurnee and Tegmark, "Language Models Represent Space and Time", October 2023](https://arxiv.org/abs/2310.02207); [Vafa et al., "Evaluating the World Model Implicit in a Generative Model", 2024](https://arxiv.org/abs/2406.03689); [LeCun, "A Path Towards Autonomous Machine Intelligence", 2022](https://openreview.net/forum?id=BZ5a1r-kVsf); [LeCun on Sora, via Futurism/Yahoo, February 2024](https://malaysia.news.yahoo.com/openais-video-generating-ai-doomed-233035802.html); [Assran et al., "V-JEPA 2", arXiv, June 2025](https://arxiv.org/abs/2506.09985); [TechCrunch on AMI Labs, January 2026](https://techcrunch.com/2026/01/23/whos-behind-ami-labs-yann-lecuns-world-model-startup/); [Heise on AMI Labs funding](https://heise.de/-11206350); [OpenAI, "Sora 2", September 2025](https://openai.com/index/sora-2/); [Bruce et al., "Genie: Generative Interactive Environments", February 2024](https://arxiv.org/abs/2402.15391); [Google DeepMind, Genie 2, December 2024](https://deepmind.google/discover/blog/genie-2-a-large-scale-foundation-world-model/); [Google DeepMind, Genie 3, August 2025](https://deepmind.google/discover/blog/genie-3-a-new-frontier-for-world-models/); [Google, Project Genie, January 2026](https://blog.google/innovation-and-ai/models-and-research/google-deepmind/project-genie/).* *Related: [Synthetic Data and Model Collapse](https://phamphutai.com/2026/10/20/synthetic-data-and-model-collapse/), and [5 Famous AI Demos That Weren't What They Looked Like](https://phamphutai.com/2026/10/18/famous-ai-demos-that-werent-what-they-looked-like/).*