Contrastive self-supervision refers to a paradigm of self-supervised learning in which a model learns representations using contrastive learning principles that maximize the similarity of positive pairs and minimize the similarity of negative pairs.
Formally, the InfoNCE loss is often used:
\[ \mathcal{L}_{i} = -\log \frac{\exp(\mathrm{sim}(\mathbf{z}_i, \mathbf{z}_i^+)/\tau)}{\sum_{k=1}^{N} \exp(\mathrm{sim}(\mathbf{z}_i, \mathbf{z}_k)/\tau)} \]
where \( \mathbf{z}_i \) and \( \mathbf{z}_i^+ \) are the representations of a positive pair, \( \mathbf{z}_k \) the representations of all examples in the batch, \( \mathrm{sim} \) a similarity function (e.g., cosine similarity), and \( \tau \) a temperature parameter.
Through this contrasting, the model learns a structured representation space that reflects semantic relationships without relying on manual annotations.
Definition:
“Contrastive self-supervision is a self-supervised learning method that learns robust and meaningful representations by maximizing the similarity between different views of the same data point and minimizing the similarity to other data points.”
Source:
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning (ICML).