Latent Diffusion: diffuse in the latent, not in pixels

13 Просмотры

Спасибо! Поделитесь с друзьями!

Вам не понравилось видео. Спасибо за то что поделились своим мнением!

Издатель Aug 7, 2026

Diffusion models generate images by denoising, but in pixel space each of the hundreds of steps
evaluates the network in about 786 thousand dimensions. That is hundreds of GPU-days to train.
Latent Diffusion Models, the paper behind Stable Diffusion, makes one substitution that fixes it.

The insight is a split. An image compresses two ways: perceptual compression, the high-frequency
detail you can barely see, and semantic compression, the actual structure. Pixel diffusion wastes
almost all its capacity modeling the imperceptible part. Latent Diffusion says, do not spend
generative capacity on that. Compress the perceptual part once with an autoencoder, and only do
the expensive generative diffusion in the small semantic latent.

In this video we walk the real data flow. We cover the autoencoder stage, an encoder and decoder
trained once with a perceptual plus adversarial loss and a KL or VQ regularizer that makes it a
VAE. We show that the latent objective, eq.2, is exactly the pixel objective eq.1 with x replaced
by z equals E of x. The score matching is identical. Only the workspace changes, and the working
dimension drops by f squared, about 50 times fewer dimensions at f equals 8. We add cross-attention
so the UNet attends to text tokens, which turns the model into a general conditional generator, and
we decode the sampled latent to an image in a single pass. Text plus this is Stable Diffusion.

Results: at matched compute, latent beats pixel diffusion by an FID gap of 38, and CelebA-HQ hits
FID 5.11, state of the art. It democratized diffusion, a few A100s instead of hundreds of V100s.
The honest limit: inference is still sequential, about 200 DDIM steps.

Who this is for: engineers and ML practitioners learning foundational AI patterns.

Paper: Rombach, Blattmann, Lorenz, Esser, Ommer, High-Resolution Image Synthesis with Latent
Diffusion Models, CVPR 2022, arXiv:2112.10752.

More paper breakdowns in the Latent Papers playlist. Subscribe for one paper a week.
#LatentDiffusion #StableDiffusion #GenerativeAI #DiffusionModels