Self-supervised learning with contrastive methods refers to a class of learning algorithms where an encoder f_θ is trained to maximize the similarity between representations of positive pairs and minimize that of negative pairs.
Formally, the InfoNCE loss is often used, which is defined for an anchor embedding z_i and its positive counterpart z_j as well as a set of negative examples z_k:
\[ \mathcal{L}_{i,j} = -\log \frac{\exp(\mathrm{sim}(z_i, z_j) / \tau)}{\sum_{k=1}^{N} \exp(\mathrm{sim}(z_i, z_k) / \tau)} \]
Here, τ is the temperature and “sim” is, for example, the cosine similarity measure.
The selection and generation of positive and negative pairs is crucial for learning success.
Newer methods like BYOL or SimSiam even dispense with explicit negative pairs and use architectural tricks to avoid collapsing solutions.
Contrastive self-supervised learning methods have proven very effective in extracting representative features from unlabeled data, which is utilized in areas such as computer vision, speech processing, and multimodal models.
Definition:
“Self-supervised learning with contrastive methods is a learning paradigm where a model learns robust representations from unlabeled data by maximizing the similarity between positive data pairs and minimizing the similarity to negative pairs.”
Source:
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning (ICML).