Diffusion models are probabilistic generative models that define a stochastic diffusion process to model data distributions.
The forward process \( q \) is a Markov process that successively adds Gaussian noise to the data:
\[ q(\mathbf{x}_t|\mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t; \sqrt{1-\beta_t}\mathbf{x}_{t-1}, \beta_t \mathbf{I}) \]
where \( \beta_t \) are small variance parameters controlling the amount of noise.
The backward process is approximated by a neural network \( p_\theta \) that learns to remove the noise and thus generate the data distribution:
\[ p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}_t, t), \Sigma_\theta(\mathbf{x}_t, t)) \]
Training minimizes a variation of the Evidence Lower Bound (ELBO) using a denoising score matching-like loss.
Diffusion models are characterized by stability during training and high quality of generated data, but are computationally intensive.
Definition:
“Diffusion models are probabilistic generative models that model data through a stochastic forward process with successive noise addition and a trained backward process for noise removal.”
Source:
Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. Advances in Neural Information Processing Systems (NeurIPS).