Contrastive Divergence is an approximate method for parameter estimation in Energy-Based Models, especially in Restricted Boltzmann Machines, which approximates the gradient of the log-likelihood by taking the difference between the data and model distributions.
Formally, the parameter update is based on the gradient
\[ \nabla_\theta \log p(x; \theta) = -\mathbb{E}_{p(h|x)}\left[\nabla_\theta E(x,h;\theta)\right] + \mathbb{E}_{p(x,h)}\left[\nabla_\theta E(x,h;\theta)\right] \]
which is approximated by replacing the second expectation with a short Gibbs sampling chain of k steps starting from the data points (CD-k):
\[ \Delta \theta \propto -\left( \mathbb{E}_{p(h|x)}\left[\nabla_\theta E(x,h;\theta)\right] - \mathbb{E}_{p(h|\tilde{x})}\left[\nabla_\theta E(\tilde{x},h;\theta)\right] \right) \]
with \( \tilde{x} \) being the samples generated by Gibbs sampling.
This method enables efficient training since only a few sampling steps are necessary to obtain good approximations.
However, the approximation introduces a bias that is tolerable in practice and can be improved by variants such as Persistent Contrastive Divergence (PCD).
Definition:
“Contrastive Divergence is an approximate learning procedure for efficient estimation of model parameters in Energy-Based Models, where the gradient of the log-likelihood is approximated by the difference between data and model distributions, with the model distribution approximated by short Gibbs sampling chains.”
Source:
Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8), 1771–1800.