Contrastive coding refers to a method of representation learning in which an embedding function \( f_\theta \) is trained so that positive pairs of data points \( (x_i, x_j^+) \) lie closer together in the latent space than negative pairs \( (x_i, x_k^-) \).
Formally, the InfoNCE loss is often used:
\[ \mathcal{L}_i = - \log \frac{\exp(\mathrm{sim}(f_\theta(x_i), f_\theta(x_j^+)) / \tau)}{\sum_{k=0}^K \exp(\mathrm{sim}(f_\theta(x_i), f_\theta(x_k)) / \tau)} \]
where \( \mathrm{sim} \) is a similarity function, e.g., cosine similarity, and \( \tau \) is a temperature variable.
The goal is to structure the representations so that semantically similar instances are grouped and dissimilar ones are separated, leading to better generalization.
Definition:
“Contrastive coding is a self-supervised learning method that produces robust and meaningful representations by maximizing the similarity of positive pairs and minimizing the similarity of negative pairs.”
Source:
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A Simple Framework for Contrastive Learning of Visual Representations. Proceedings of the 37th International Conference on Machine Learning (ICML).