Self-Supervised Learning is a learning paradigm in which a model uses tasks generated from the data itself to learn useful representations.
These so-called pretext tasks are defined, in which parts of the input are predicted or reconstructed, e.g., filling in missing image regions or predicting the next word in the text.
This approach enables pretraining without explicit labels, which can then be fine-tuned for specific tasks.
Self-Supervised Learning has the advantage of effectively utilizing large amounts of unlabeled data and improving generalization.
Well-known approaches include Contrastive Predictive Coding, Masked Language Models like BERT, or the prediction of image transformations.