Skip to main content

Day 14: The Generative Zoo - Diffusion Models

Day 14: The Generative Zoo - Diffusion Models​

Objective​

Understand the high-level concept behind Diffusion Models, the current state-of-the-art for high-fidelity image generation, and how they differ from GANs.

Core Concepts​

  • The Problem with GANs:

    • GANs are powerful but suffer from unstable training and mode collapse. The adversarial dynamic can be difficult to balance.

  • Diffusion Models: The Big Idea:

    • Instead of trying to generate an image in one shot from a random vector, Diffusion Models learn to reverse a process of gradually adding noise.
    • It's a two-step process:
      1. The Forward Process (Fixed): Take a real image from the dataset and slowly add a little bit of Gaussian noise at each time step t. After many steps (T), the image becomes indistinguishable from pure random noise. This process is mathematically defined and does not involve any learning.
      2. The Reverse Process (Learned): This is where the magic happens. A neural network is trained to undo one step of the noising process. Its job is to take a noisy image at step t and predict the noise that was added to get to that state. By subtracting this predicted noise, it can produce a slightly less noisy image from step t-1.

  • How Generation Works:

    • You start with a pure random noise image (the same kind of image that the forward process ends with).
    • You feed this noise into your trained neural network.
    • The network predicts the noise present in the image. You subtract a small amount of this predicted noise.
    • You take the resulting, slightly-less-noisy image and feed it back into the network.
    • You repeat this "denoising" process many times. Each step refines the image, gradually transforming the pure noise into a clean, coherent image.

  • The Denoising Network:

    • The neural network at the heart of a diffusion model is typically a U-Net.
    • A U-Net is a type of encoder-decoder architecture with "skip connections" that allow information from the encoder's downsampling path to be passed directly to the decoder's upsampling path. This helps the network preserve fine-grained details, which is crucial for predicting the exact noise pattern.

🧠 Math & Stats Focus: Gaussian Distributions & Denoising Score Matching​

  • Gaussian Noise: The noise added at each step in the forward process is sampled from a simple Gaussian (Normal) distribution, N(0, I). The variance of the noise is increased slightly at each step according to a pre-defined "variance schedule". This makes the math of the process tractable.

  • The Model's Objective: The neural network is not trained to predict the image at step t-1 directly. It is trained to predict the noise ε that was added to the image at step t.

    • The loss function is typically the Mean Squared Error (MSE) between the true noise ε (which is known, because we added it) and the predicted noise ε_θ from the model θ.
    • Loss = || ε - ε_θ(x_t, t) ||²
    • This is an example of "score matching," where the model learns the gradient of the data distribution's log-probability (the "score").

  • Why is this better than GANs?

    • Stable Training: The training process is much more stable than GANs. You are just training a network to predict noise with a simple MSE loss, which is a standard supervised learning problem. There is no unstable minimax game.
    • High-Quality & Diverse Samples: Diffusion models have been shown to produce higher-quality and more diverse images than GANs, with less risk of mode collapse. The main downside is that the iterative generation process can be slower than a GAN's single-pass generation.

📜 Key Research Paper​

  • Paper: "Denoising Diffusion Probabilistic Models" (Ho, Jain, & Abbeel, 2020)

  • Link: https://arxiv.org/abs/2006.11239

  • Contribution: While the core ideas of diffusion had been around for a while, this paper simplified the approach and demonstrated that diffusion models could achieve very high-quality image generation, rivaling and even surpassing GANs. This work, along with follow-ups, directly led to the development of models like DALL-E 2 and Stable Diffusion.

  • For Stable Diffusion: "High-Resolution Image Synthesis with Latent Diffusion Models" (Rombach et al., 2022)

  • Link: https://arxiv.org/abs/2112.10752

  • Contribution: Introduced Latent Diffusion. Instead of running the expensive diffusion process in the high-dimensional pixel space, they first use an autoencoder to compress the image into a smaller "latent space." The diffusion/denoising process happens in this much smaller space, making it vastly more efficient and enabling the generation of very high-resolution images on consumer hardware. This is the core idea behind Stable Diffusion.

💻 Project: Generate Images with a Pre-trained Diffusion Model​

Training a diffusion model from scratch is computationally very intensive. The best way to get started is to use a pre-trained model via the Hugging Face diffusers library.

  1. Install Libraries: pip install diffusers transformers accelerate.
  2. Load a Pre-trained Pipeline:
    • from diffusers import DDPMPipeline
    • pipeline = DDPMPipeline.from_pretrained("google/ddpm-cat-256") (A model pre-trained to generate pictures of cats).
  3. Generate an Image:
    • image = pipeline().images[0]
  4. Display the Image: Use matplotlib or the PIL library to display the generated image.
  5. Explore Stable Diffusion: Go a step further and try a text-to-image pipeline with Stable Diffusion.
    • from diffusers import StableDiffusionPipeline
    • pipeline = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
    • prompt = "a photograph of an astronaut riding a horse on mars"
    • image = pipeline(prompt).images[0]
    • Try different prompts and see what you can create!

✅ Progress Tracker​

  • I can describe the two main processes of a diffusion model (forward noising, reverse denoising).
  • I can explain the high-level difference between how a GAN generates an image and how a diffusion model generates an image.
  • I understand why diffusion models are generally more stable to train than GANs.
  • I have used a pre-trained diffusion model to generate an image from a text prompt.