Contrastive representation learning is a method of self-supervised learning that aims to learn an embedding function \( f: \mathcal{X} \to \mathbb{R}^d \) such that similar inputs \( x_i, x_j \) lie close to each other in the feature space, while dissimilar inputs are farther apart.
Formally, the InfoNCE loss is often used, defined as:
\[ L_i = -\log \frac{\exp(\mathrm{sim}(f(x_i), f(x_j^+)) / \tau)}{\sum_{k=1}^{N} \exp(\mathrm{sim}(f(x_i), f(x_k)) / \tau)} \]
Here, \( x_j^+ \) denotes a positive example for \( x_i \), \( x_k \) negative examples, \( \mathrm{sim} \) a similarity function (e.g., cosine similarity), and \( \tau \) a temperature constant.
The learning process optimizes the function \( f \) so that positive pairs are closer and negative pairs are farther apart.
Contrastive learning is the basis of many modern self-supervised learning methods such as SimCLR, MoCo, or BYOL, which enable learning powerful representations without extensive labeling.
Definition:
“Contrastive representation learning is a self-supervised learning approach that produces a meaningful embedding by maximizing the similarity between positive data pairs and minimizing the similarity between negative pairs.”
Source:
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A Simple Framework for Contrastive Learning of Visual Representations. Proceedings of the 37th International Conference on Machine Learning (ICML).