How Image Generation Works

Start with noise and repeatedly remove a little of it. Diffusion is a different mechanism from text generation, and the difference explains the controls.

Text generation builds forward, one token at a time. Image generation works nothing like that.

It starts with pure random noise and repeatedly removes a little of it, guided by your prompt, until an image emerges. Twenty to fifty passes, each one slightly less noisy than the last.

The training insight

The clever part is how such a model is trained, and it inverts the problem.

Take a real image. Add a small amount of noise. Train a network to predict the noise you added. Repeat across many noise levels — from barely perceptible to complete static.

The network becomes a noise predictor. And a noise predictor is a denoiser: subtract what it predicts and you get something cleaner.

Generation runs that in reverse. Start from pure noise, ask the network what noise it sees, subtract a portion, repeat. Because the network learned this on real images, the thing that emerges from noise looks like a real image.

Each step is a small, easy prediction. The difficulty is spread across many steps rather than concentrated in one — which is the same reason chain-of-thought helps language models, arrived at from a completely different direction.

Where the prompt enters

The denoiser is conditioned on text. Your prompt is encoded into embeddings, and the network attends to them at every step through cross-attention.

So the question the network answers is not “what noise is here” but “what noise is here, given that this should become a photograph of a cat on a windowsill.” Every denoising step is nudged toward matching the description.

Classifier-free guidance amplifies this. At each step, run the network twice — once with the prompt, once without — and push away from the unconditional prediction. A guidance scale controls how hard. Low values follow the prompt loosely and look natural; high values follow it aggressively and eventually look oversaturated and strange. This is the single most important knob, and it is the direct analogue of temperature in text.

Latent diffusion

Denoising at full pixel resolution is expensive. The dominant optimization is to work in a compressed space instead.

An autoencoder compresses the image to a much smaller latent representation. Diffusion happens there. A decoder expands the final latent back to pixels.

The saving is large — a 512×512 image might diffuse in a 64×64 latent — and it is what made high-resolution generation practical on ordinary hardware. It also explains a class of artifact: fine detail is reconstructed by the decoder rather than diffused directly, which is part of why small text and precise textures come out mangled.

The controls

Steps. More denoising passes, more refinement. Returns diminish quickly, and past a point extra steps only cost time.

Guidance scale. Prompt adherence versus naturalness, as above.

Seed. The initial noise. Same seed plus same prompt plus same settings reproduces the same image — which is what makes iteration possible.

Negative prompt. A second condition to steer away from. Mechanically it replaces the unconditional prediction in guidance, which is why it works at all.

Image-to-image. Start from a real image with partial noise added rather than pure noise. The amount of noise controls how far the result drifts from the original.

Inpainting. Denoise only a masked region, keeping the rest fixed.

What it is bad at

The failures are characteristic and mostly structural.

Text in images. Improving, historically terrible. The model has no notion of letters as discrete symbols, and latent compression damages fine strokes.

Counting and quantity. “Three cats” is unreliable for the same reason vision models cannot count — nothing individuates objects.

Hands and anatomy. Highly variable structures with strong constraints, poorly represented by a process that works globally rather than skeletally.

Compositional relationships. “A red cube on top of a blue sphere” frequently swaps the attributes. Guidance conditions the whole image, not bound object-attribute pairs.

Precise instruction following. The prompt is a soft nudge distributed across steps, not a specification.

Note on transformers

Recent systems increasingly replace the older convolutional denoiser with a transformer operating on latent patches — the same patch-token idea from how models see images. The diffusion process is unchanged; only the architecture of the denoiser differs. Scaling behaviour has been better, which is why the shift happened.

What to remember

• Diffusion starts from noise and removes it repeatedly, typically 20–50 steps — not token-by-token like text.
• Training teaches the network to predict added noise; generation runs that in reverse.
• The prompt conditions every step; guidance scale trades adherence against naturalness.
• Latent diffusion compresses before denoising, which made it practical and explains fine-detail artifacts.
• Weak at text, counting, anatomy, and binding attributes to objects.

Next: Multimodal Embeddings