# How Image Generators Work: Diffusion Explained > How AI image generators turn random noise into pictures: denoising, text encoders, latent diffusion, guidance, why hands failed, and the copyright fights. - Author: Tai Pham Phu - Published: 20 Oct 2026 - Tags: How AI Works, LLM, Applied AI, Image Generation, Diffusion - URL: https://phamphutai.com/2026/10/20/how-image-generators-work-diffusion/ *Part 7 of How AI Works: Advanced. In [part 6](https://phamphutai.com/2026/10/20/how-ai-sees-vision-transformers-and-clip/) we saw how AI reads images. This part goes the other way: how AI makes them. If tokens or attention are new to you, start with the basics first.* On 29 August 2022, the Colorado State Fair published its fine arts results. In the emerging artist division, under "Digital Arts / Digitally-Manipulated Photography", first place went to Jason Allen for a piece called *Théâtre D'opéra Spatial*. Allen had made it with Midjourney, an AI image generator. He had entered it as "Jason Allen via Midjourney", and before the results he had posted online about creating hundreds of images and spending weeks fine-tuning and curating. When the win became public, many artists were furious. The argument spread far beyond Colorado: is this art, is it cheating, and who made it? The second chapter came from the US Copyright Office. Allen applied to register the work on 21 September 2022. He told the Office he had typed revisions and text prompts **at least 624 times** to get the first version, then cleaned it up in Photoshop and enlarged it with an AI upscaler. The Office asked him to exclude the parts Midjourney made. He refused. On 5 September 2023, the Office's Review Board refused registration, finding that the work contained more than a small amount of AI-generated material that had to be disclaimed. Allen sued in federal court in Colorado in September 2024. At the time of writing (October 2026), I couldn't find a ruling in that case. 624 prompts is a lot of work. But what was the machine doing with each one? Midjourney has never published how its models work. The open, well-documented systems that most image generators descend from use a method called **diffusion**, and it's stranger and simpler than you'd expect. --- ## The problem: a picture is a lot of numbers To a computer, an image is a grid of numbers. A 512 by 512 colour picture has 512 × 512 pixels, each with three numbers for red, green and blue. That's **786,432 numbers**. To make a picture, a model has to choose all of those numbers so that together they form a coherent scene, where the left side agrees with the right side and the shadows agree with the light. Diffusion solves this with a trick: don't make the picture in one go. Start from noise and clean it up, a little at a time. --- ## The trick: learn to remove noise The idea first appeared in a 2015 paper by Jascha Sohl-Dickstein and colleagues. It became practical in 2020, when Jonathan Ho, Ajay Jain and Pieter Abbeel at UC Berkeley published **"Denoising Diffusion Probabilistic Models"**, usually called DDPM. Their samples beat the best published scores on a standard image benchmark (CIFAR-10) at the time, and most of today's image generators trace back to that paper. Here's how it works, in two halves. **Training: add noise, ask the model to guess it.** 1. Take a real photo from the training set. 2. Add a random amount of noise to it, like TV static sprinkled over the picture. Sometimes a little, sometimes so much the photo disappears. 3. Show the noisy picture to a neural network and ask: *what noise was added?* 4. Compare its guess with the noise you actually added, and nudge the network to guess better next time. You never run out of exercises, and you always know the right answer, since you added the noise yourself. In the DDPM paper, the noise was added over 1,000 small steps, from a clean picture to pure static. **Generation: start with static, remove a bit at a time.** 1. Start with pure random noise. No picture at all. 2. Ask the trained network: *what noise is in this?* 3. Subtract a small part of its guess. The result is slightly less noisy. 4. Repeat, many times. After enough rounds, a picture appears that was never in the training data.
TRAINING: ADD NOISE, STEP BY STEP (EASY, NO LEARNING NEEDED) clean image a little noise half noise mostly noise pure noise GENERATION: REMOVE A LITTLE NOISE EACH STEP (THIS IS WHAT THE MODEL LEARNS) Start from fresh static on the right and denoise leftward, guided by the prompt.
Training adds noise to real pictures and teaches the model to guess it. Generation runs the other way, starting from fresh static.
A useful analogy is a photo restorer cleaning a badly damaged print. Each pass removes some grime and reveals a bit more of the scene. Here's where the analogy breaks, and it's the most important part. **When you generate, there is no original photo under the noise.** The static is fresh and random. The model isn't uncovering a hidden picture. It's inventing one that would make sense under the noise, based on millions of photos it has seen. Different starting static gives a different picture, which is why the same prompt gives you four different images. The Ho paper also noticed something about the order things happen in: **large features appear first, details last.** In the first, very noisy steps, the model is effectively choosing the layout: where the sky goes, where the big shapes sit. In the last steps it's choosing textures, edges and small details. Keep that in mind, because it helps explain the hands problem later. --- ## Text conditioning: how words steer the noise Plain diffusion makes *some* plausible picture. To make the picture you asked for, the model needs to read your prompt. This is called **conditioning**: the denoiser gets the prompt as an extra input at every step. The prompt goes through a **text encoder** first: a model that turns words into numbers that capture their meaning. Two kinds are common: - **CLIP.** OpenAI's 2021 model, trained on 400 million image and caption pairs to put matching pictures and text close together. We covered it in [part 6](https://phamphutai.com/2026/10/20/how-ai-sees-vision-transformers-and-clip/). Its text half is useful here because it already "knows" which words go with which visual ideas. The original Stable Diffusion used a frozen CLIP text encoder. - **Language model encoders like T5.** Google's Imagen paper (2022) used T5, a model trained only on text. One of its findings was that making the text encoder bigger improved image quality and prompt matching more than making the image model bigger. Inside the denoiser, the link between picture and prompt is **cross-attention**: each area of the image being cleaned can "look at" the prompt's words and pull in what's relevant. The area becoming the sky attends to "sunset". The area becoming a face attends to "smiling old man". It's the same attention idea used in language models, applied between two different kinds of data. --- ## Latent diffusion: why it fits on a gaming card Early diffusion models worked directly on pixels. Every step processed all 786,432 numbers, and you needed many steps. That was slow and expensive. In December 2021, Robin Rombach and colleagues (Ludwig Maximilian University of Munich and Runway) published **"High-Resolution Image Synthesis with Latent Diffusion Models"**. The idea: don't denoise pixels. Denoise a **compressed version** of the picture. 1. First, train a separate **autoencoder**: a pair of networks that squeeze an image into a much smaller grid of numbers (the **latent**) and expand it back, keeping what our eyes care about. 2. Run the whole diffusion process inside that small latent space. 3. At the very end, use the decoder half to expand the result into a full-size image. In the original Stable Diffusion script, the autoencoder shrinks each side by 8. A 512 × 512 image becomes a 64 × 64 grid with 4 numbers per cell: **16,384 numbers instead of 786,432, about 48 times fewer.** (That's my arithmetic from the script's default settings.) That's the model that went public as **Stable Diffusion** on 22 August 2022, a week before the Colorado results, made by the Munich group with Runway, Stability AI and LAION, trained on 512 × 512 images from a subset of the LAION-5B dataset, with open weights anyone could download. Its README said it ran on a GPU with at least 10 GB of memory. For the first time, a strong image generator could run on a gaming PC instead of a data centre.
1 · READ THE PROMPT your prompt "a sun over a mountain" text encoder (CLIP, T5) words become numbers steers every step 2 · DENOISE IN A SMALL "LATENT" SPACE random noise 64 × 64 × 4 numbers denoiser (U-Net or transformer) guess the noise WITH the prompt guess the noise WITHOUT it push toward the difference × 7.5 remove a little noise, repeat ~50 times 3 · DECODE decoder latent → pixels 512 × 512 image Numbers are the original Stable Diffusion v1 script defaults (CompVis, 2022).
Latent diffusion: read the prompt, denoise a small compressed grid many times with guidance, then decode it into pixels.
Analogy: an architect works on a floor plan, not a full-size building. The plan is small, quick to change and holds the decisions that matter. Where it breaks: a floor plan is designed by people to be readable. The latent is learned by a network and isn't readable by anyone. It also throws away detail the autoencoder thought you wouldn't notice. Remember that too. --- ## Guidance: the volume knob on your prompt If you've used an open image tool, you've seen a setting called **CFG scale** or **guidance scale**, often defaulting to around 7. Here's what it does, in plain words. It comes from a 2022 paper by Jonathan Ho and Tim Salimans called **classifier-free guidance**. During training, the prompt is sometimes left out, so the same model learns to denoise both with and without a prompt. Then, at each generation step: 1. The model guesses the noise **with** your prompt. 2. It also guesses the noise **without** any prompt. 3. The difference between the two guesses is "what your prompt adds". 4. The model takes the no-prompt guess and adds that difference multiplied by the guidance scale. A scale of 1 means "just use the prompt normally". A scale of 7.5, the default in the original Stable Diffusion script, means "push 7.5 times as hard in the direction the prompt points". As a formula in words: *final guess = no-prompt guess + scale × (prompt guess − no-prompt guess).* The paper describes this as a trade-off between **fidelity and diversity**. Turn it up and images match the prompt more closely but vary less from one run to the next. Turn it down and you get more variety and looser interpretations. Google's Imagen paper found that very high guidance makes images "highly saturated and unnatural", like a song with the bass turned all the way up. One practical side effect: guidance usually means two passes per step instead of one. Steps × 2 is a big part of why image generation costs what it does. --- ## Why hands and text used to fail For a couple of years, the easiest way to spot an AI image was to count fingers or read the sign in the background. Both have improved a lot, and the reasons they failed are a good tour of the system's weak spots. ### Text: the model never saw the letters This one is well documented. When OpenAI described DALL-E 3 in 2023, they wrote that its text rendering was unreliable, with missing or extra characters, and suspected the T5 text encoder: the model sees **tokens that stand for whole words** and has to work out which letters to draw. That's the same tokenizer problem behind chatbots miscounting the R's in "strawberry", which [part 8](https://phamphutai.com/2026/10/20/glitch-tokens-solidgoldmagikarp/) digs into. A 2022 Google paper by Rosanne Liu and colleagues tested this directly. Image models with **character-aware** text encoders, which see individual letters, spelled far better than "character-blind" ones, with gains of more than 30 accuracy points on rare words. Compression adds to the problem. Small text is fine detail, exactly what a heavily compressed latent is most likely to blur. ### Hands: lots of variety, little attention There's no single paper that proves why hands failed, so treat these as the likely reasons researchers and practitioners point to, not settled science: - **Hands are small** in most training photos, so they get few pixels and even fewer latent cells. - **Hands vary enormously.** Fingers bend, overlap, hold things and hide behind each other. The model sees endless partial views and few clean, countable ones. - **Captions rarely mention hands.** A photo is captioned "woman drinking coffee in a café", not "woman whose right hand shows four fingers and a thumb around a cup". - **Counting is global, drawing is local.** Remember that details come late in denoising. By then, each patch is filling in plausible finger texture, and nothing forces the patches to agree on how many fingers there are. ### What changed Several things, together: - **Better captions.** DALL-E 3's big idea was retraining on detailed, machine-written descriptions of every image. Its final model trained on 95% synthetic captions, which made it follow prompts much more closely. - **Stronger text encoders**, including large language models. - **Transformers instead of the older U-Net denoiser**, which scale better with more compute (more below). - **Much more compute and data.** Stability AI's Stable Diffusion 3 paper (2024) specifically claimed better typography from its new architecture. --- ## Newer approaches, briefly Diffusion has kept evolving, and not every image tool uses it anymore. The main threads: - **Diffusion transformers.** In 2022, William Peebles and Saining Xie replaced the U-Net denoiser with a transformer working on patches of the latent, and found that more compute kept giving better images. Stable Diffusion 3, below, is one of the models built this way. - **Flow matching, or rectified flow.** Instead of a noisy, winding path from static to picture, the model learns a straighter path between them, which can need fewer steps. Stable Diffusion 3 (Esser and colleagues, 2024) uses this, along with a transformer that keeps separate weights for text and image. - **Few-step distillation.** A big, slow model teaches a smaller, faster one to jump to the answer in fewer steps. Stability AI's Adversarial Diffusion Distillation paper (2023) reported good images in **1 to 4 steps** instead of dozens. Same idea as distillation in [part 4](https://phamphutai.com/2026/10/20/quantization-and-distillation/). - **Autoregressive image generation.** When OpenAI launched image generation inside GPT-4o on 25 March 2025, its system card said that unlike DALL-E, which works as a diffusion model, 4o image generation is an **autoregressive** model built into ChatGPT. In other words, closer to how a chatbot produces text. OpenAI hasn't published the full design. Many commercial tools, Midjourney included, don't publish their architectures at all. If someone tells you exactly how a closed model works, ask how they know. --- ## The copyright debates, stated plainly Image generators sit in the middle of two separate legal arguments. Both are unsettled, and reasonable people disagree. I'm not a lawyer, so here are the facts and the main positions, as of October 2026. ### Question 1: who owns an AI-generated image? In the US, copyright needs a human author. In January 2025, the Copyright Office published a report on AI and copyrightability concluding that, with current technology, "prompts alone do not provide sufficient human control" to make the user the author. Human work still counts: your own edits, your selection and arrangement of images, and your own material that stays visible in the result can be protected. That's why the Office offered to register Allen's Photoshop changes but not the Midjourney image underneath. The courts have backed the core rule so far. In *Thaler v. Perlmutter*, about an image listed as made solely by an AI system, an appeals court confirmed in March 2025 that copyright requires a human author, and the US Supreme Court declined to take the case in March 2026. That case didn't decide how much human input is enough. Allen's lawsuit is one of the first to ask exactly that, and at the time of writing it was still waiting for a decision. **Allen's side:** hundreds of prompts, choices and edits are creative control, like a photographer choosing settings and shots. **The Office's side:** prompts mostly describe ideas, and the model decides the actual visual expression in ways the user can't predict. ### Question 2: is training on copyrighted images allowed? Models like Stable Diffusion learned from huge collections of images gathered from the web. LAION-5B, the dataset behind it, has 5.85 billion image and text pairs, collected without asking each creator. The main cases at the time of writing: - **Andersen v. Stability AI** (US). A group of artists sued Stability AI, Midjourney, DeviantArt and Runway in 2023. In August 2024, the judge let their copyright and trademark claims go forward while dismissing others. That's a ruling that the case can continue, not a finding that anyone infringed. Reports in 2026 put a possible trial in 2027. - **Getty Images v. Stability AI** (UK). Judgment came on 4 November 2025. Getty dropped its main claim about training during the trial. The judge rejected Getty's remaining copyright claim, finding the model's weights don't store copies of Getty's images. Getty won limited trademark points over fake Getty watermarks that early versions sometimes produced. Getty also has a separate case in the US. - **Disney and Universal v. Midjourney** (US). Filed in June 2025. Warner Bros. Discovery later brought a similar suit, and the cases were combined. It was still pending at the time of writing. **The AI companies' position:** the model learns general patterns from images, the way a person learns by looking, and doesn't keep copies, so training is fair use. **The creators' position:** their work was copied without permission or payment to build products that compete with them, and the models can imitate their style and sometimes their specific characters. --- ## What this means in practice If you build with image generators or buy tools that use them: - **Don't assume you own the output.** In the US, a purely generated image may have no copyright at all, which means competitors may be free to copy it. If ownership matters, add real human work on top and keep a record of it. For anything important, ask a lawyer. - **Read the tool's terms.** Check who owns outputs, what you're allowed to do with them, and whether the vendor offers any legal protection if a claim comes up. - **Check text and hands every time.** They're much better now, but they're still the first things to go wrong. Never ship a logo, label or menu without reading every letter. - **Steps and guidance are speed and quality knobs.** More steps cost more and take longer. Very high guidance gives literal but stiff images. Distilled "turbo" models trade some quality for huge speed gains, which matters if you're generating at volume. - **Disclose AI use when it matters.** A short label is cheaper than a controversy. - **Be careful with named characters and living artists' styles.** That's exactly where the lawsuits are. --- ## Key ideas - **Diffusion models learn one skill: guessing the noise in a noisy image.** Generation starts from pure static and removes a little predicted noise, many times over. - **There's no hidden original under the static.** The model invents a picture that fits, so new static means a new image. - **A text encoder (CLIP, T5 or a language model) turns your prompt into numbers**, and cross-attention lets each part of the image look at them at every step. - **Latent diffusion runs the whole process on a compressed version** (about 48 times fewer numbers in the original Stable Diffusion), which is why it could run on a gaming PC. - **Guidance scale is a volume knob for the prompt:** higher means closer to your words and less variety. - **Text failed because the encoder saw tokens, not letters; hands failed for less settled reasons.** Better captions, encoders and bigger transformer models fixed most of it. - **Copyright is unsettled on two fronts:** whether outputs can be owned, and whether training on copyrighted images is legal. *Sources: [Colorado State Fair 2022 fine arts results](https://coloradostatefair.com/wp-content/uploads/2022/08/2022-Fine-Arts-First-Second-Third.pdf); [Vice on Allen's win, Aug 2022](https://www.vice.com/en/article/an-ai-generated-artwork-won-first-place-at-a-state-fair-fine-arts-competition-and-artists-are-pissed/); [US Copyright Office Review Board decision on Théâtre D'opéra Spatial, Sept 2023](https://www.copyright.gov/rulings-filings/review-board/docs/Theatre-Dopera-Spatial.pdf); [Allen v. Perlmutter complaint, D. Colo., Sept 2024](https://business.cch.com/ipld/AllenPerlmutterComp20240926.pdf); [Sohl-Dickstein et al., 2015](https://arxiv.org/abs/1503.03585); [Ho, Jain & Abbeel, DDPM, 2020](https://arxiv.org/abs/2006.11239); [Radford et al., CLIP, 2021](https://arxiv.org/abs/2103.00020); [Saharia et al., Imagen, 2022](https://arxiv.org/abs/2205.11487); [Rombach et al., Latent Diffusion Models, 2021](https://arxiv.org/abs/2112.10752); [CompVis Stable Diffusion repository and txt2img defaults](https://github.com/CompVis/stable-diffusion); [Stability AI, Stable Diffusion public release, Aug 2022](https://stability.ai/news-updates/stable-diffusion-public-release); [Ho & Salimans, Classifier-Free Diffusion Guidance, 2022](https://arxiv.org/abs/2207.12598); [Betker et al., Improving Image Generation with Better Captions (DALL-E 3), 2023](https://cdn.openai.com/papers/dall-e-3.pdf); [Liu et al., Character-Aware Models Improve Visual Text Rendering, 2022](https://arxiv.org/abs/2212.10562); [Peebles & Xie, Diffusion Transformers, 2022](https://arxiv.org/abs/2212.09748); [Esser et al., Stable Diffusion 3 / rectified flow, 2024](https://arxiv.org/abs/2403.03206); [Sauer et al., Adversarial Diffusion Distillation, 2023](https://arxiv.org/abs/2311.17042); [OpenAI, GPT-4o native image generation system card addendum, Mar 2025](https://cdn.openai.com/11998be9-5319-4302-bfbf-1167e093f1fb/Native_Image_Generation_System_Card.pdf); [US Copyright Office, Copyright and AI Part 2: Copyrightability, Jan 2025](https://www.copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-2-Copyrightability-Report.pdf); [Skadden on Thaler v. Perlmutter, Mar 2025](https://www.skadden.com/insights/publications/2025/03/appellate-court-affirms-human-authorship); [Copyright Lately on the Thaler cert denial and Allen status, 2026](https://copyrightlately.com/thaler-is-dead-ai-copyright-questions/); [Schuhmann et al., LAION-5B, 2022](https://arxiv.org/abs/2210.08402); [The Art Newspaper on the Andersen ruling, Aug 2024](https://www.theartnewspaper.com/2024/08/14/artists-win-small-victory-artificial-intelligence-copyright-case); [Getty Images v Stability AI, \[2025\] EWHC 2863 (Ch)](https://www.judiciary.uk/judgments/getty-images-v-stability-ai/); [AIPPI summary of the Getty judgment](https://www.aippi.org/news/getty-images-v-stability-ai-uk-high-court-finds-no-secondary-copyright-infringement-in-landmark-genai-ruling/); [Plagiarism Today on Disney and Universal v. Midjourney, June 2025](https://www.plagiarismtoday.com/2025/06/11/disney-and-universal-sue-midjourney-call-it-a-bottomless-pit-of-plagiarism).* *Related: [How AI Sees: Vision Transformers and CLIP](https://phamphutai.com/2026/10/20/how-ai-sees-vision-transformers-and-clip/), [Glitch Tokens: The Strange Words That Broke GPT-3](https://phamphutai.com/2026/10/20/glitch-tokens-solidgoldmagikarp/), and [Famous AI Demos That Weren't What They Looked Like](https://phamphutai.com/2026/10/18/famous-ai-demos-that-werent-what-they-looked-like/).*