Diffusion Models: How Stable Diffusion Draws

A diffusion model learns to remove noise in small steps: the forward process drowns an image in Gaussian noise, the network predicts that noise, and generation is denoising from pure noise. We cover latent diffusion, guidance and speeding up to 1–4 steps.

Intuition: easy to spoil, fix a little at a time

Learning to “draw an image from nothing” in one jump is a monstrously hard task: both VAEs and GANs broke on it, each in their own way. Diffusion sneaks up smarter: spoiling is easy! Anyone can add a little noise to an image. And since spoiling is easy, you can generate as many training pairs “slightly noisier image → slightly cleaner” as you like — and teach the network a small repair step. One small step is a simple task. And a chain of dozens of small steps turns pure noise into an image.

Forward: drowning the image in noise

The forward process over T steps gradually mixes Gaussian noise into the image x₀. Conveniently, you can jump straight to any step t with one formula:

Here ᾱ_t is the noise schedule: the fraction of “signal” left at step t. At the start ᾱ₀ = 1 (clean image), at the end ᾱ_T ≈ 0 (pure noise). The curve shape is chosen (linear, cosine schedule) so that information is destroyed evenly, not all at once.

Reverse: the network predicts noise

The reverse process trains a neural net (historically a U-Net, in newer models a transformer) on a noisy image x_t and step index t to predict the noise ε that was mixed in. Not the image — the noise! The loss is almost indecently simple:

No minimax game, no fragile equilibrium — ordinary regression that trains as stably as a classifier. Knowing the predicted noise, the forward formula easily yields an estimate of the clean image and a step “back”: a little less noise, a little more image. Generation is a start from pure noise x_T ∼ N(0, 1) and T denoising steps.

Text-to-image: cross-attention and guidance

For the model to draw “a cat in a spacesuit” rather than anything, it is conditioned on text. The prompt is run through a text encoder (CLIP or T5) into embeddings, and at every denoising step the network peeks at them via cross-attention: query — from pixels/latent, key and value — from prompt tokens (the same mechanism as in attention).

The second lever is classifier-free guidance (CFG). At every step the model makes two noise predictions: with the prompt and without (unconditional), then extrapolates toward the conditional:

The strength w is a “how much to obey the prompt” slider: at w = 1 guidance is off, at w ≈ 5–8 the image closely follows the text, and when overdone the result is “overcooked” — acid colors, burned contrast, lost diversity.

Latent diffusion and speeding up

Running diffusion directly in 512×512×3 pixels is expensive. Stable Diffusion does diffusion in the VAE latent: the encoder compresses the image into a 64×64×4 tensor — about 48× fewer numbers — the whole diffusion happens there, and only at the end the VAE decoder expands the latent into pixels. That is latent diffusion (LDM) — the main reason Stable Diffusion runs on an ordinary GPU.

Next they speed up the number of steps. DDIM makes the process deterministic and allows larger steps — 20–50 steps instead of 1000. Distillation (LCM, SDXL-Turbo, adversarial distillation) teaches a student model to walk the teacher's path in 1–4 steps — the interactive shows what you pay for large jumps. And architecturally the field has shifted from U-Net to DiT (diffusion transformer) — a transformer over latent patches: it scales better and has become the standard for new models (SD3, Flux, Sora).

All interview questions in this section: Multimodal Models — interview questions