Contrastive learning is a paradigm of representation learning that aims to learn an embedding function f: X → ℝ^d such that similar examples (positive pair) lie close to each other in the feature space and different examples (negative pair) lie far apart.
Formally, the InfoNCE loss is often used, which for a positive pair (x, x^+) and a set of negative examples {x^-_k} is defined as:
L = - \log \frac{\exp(\mathrm{sim}(f(x), f(x^+)) / \tau)}{\exp(\mathrm{sim}(f(x), f(x^+)) / \tau) + \sum_k \exp(\mathrm{sim}(f(x), f(x^-_k)) / \tau)}
Here, \mathrm{sim}(\cdot, \cdot) is a similarity function, e.g., the dot product, and \tau > 0 is a temperature parameter for scaling.
By minimizing this loss, the model is encouraged to cluster positive pairs and separate negatives, resulting in meaningful representations.
Contrastive learning forms the basis of many modern self-supervised methods such as SimCLR (Chen et al., 2020) and MoCo (He et al., 2020), which achieve state-of-the-art results in image processing.
Definition:
“Contrastive learning is a learning procedure in which an embedding is trained so that similar data points lie closer together in the feature space than dissimilar ones, by explicitly using positive and negative pairs.”
Source:
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning (ICML).