Contrastive learning methods are self-supervised learning techniques that train an embedding function f: X → R^d such that for an anchor example x_i, the representation f(x_i) is closer to that of positive examples x_j^+ than to negative examples x_k^-.
This is often formally implemented through a contrastive loss such as the NT-Xent Loss (Normalized Temperature-scaled Cross Entropy Loss), which maximizes or minimizes similarities for a batch of examples.
The method is based on the assumption that similar data points (e.g., different augmentations of the same image) should have semantically related representations, while dissimilar examples must be distinguished.
Modern approaches like SimCLR (Chen et al., 2020) use strong data augmentations and large batch sizes, while methods like MoCo (He et al., 2020) utilize a queue for negative examples.
Definition:
“Contrastive learning methods are self-supervised learning techniques that learn a meaningful representation of the data by maximizing the similarity between positive example pairs and minimizing the similarity to negative examples.”
Source:
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning (ICML).