Self-supervised learning with masking refers to a training paradigm in which a model f_θ is trained to predict masked parts of the input data x_{mask} based on the unmasked context x_{context}.
Formally, this can be described as minimizing a reconstruction loss:
\[ \mathcal{L}(\theta) = \mathbb{E}_{x \sim D} \left[ \ell (f_\theta(x_{context}), x_{mask}) \right] \]
where \( \ell \) is an appropriate loss function, e.g., cross-entropy for discrete tokens or MSE for continuous data.
Well-known models in NLP include BERT (Devlin et al., 2019), which uses masked language modeling for context representations.
In the field of computer vision, the concept was extended by Masked Autoencoders (MAE), which mask large image regions and have the model reconstruct them.
These approaches are effective for learning self-supervised representations that can be used in downstream tasks with little additional training.
Definition:
“Self-supervised learning with masking is a learning paradigm in which a model is trained to specifically reconstruct masked parts of the input data based on the unmasked context in order to learn robust and generalizable representations.”
Source:
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805.