Self-supervised learning with data augmentation refers to a class of learning methods where a model learns to form robust and generalizable representations through artificially generated variations of the input data.
Formally, an input datum \( x \) is transformed by a stochastic augmentation function \( t \sim \mathcal{T} \), such that \( \tilde{x} = t(x) \) is created.
The goal is to train an embedding function \( f_\theta \) that produces similar representations for two augmentations of the same original, i.e., \( f_\theta(\tilde{x}_i) \approx f_\theta(\tilde{x}_j) \), while keeping representations of different data points distinct.
This is often realized through contrastive losses (e.g., InfoNCE) or other distance metrics.
Data augmentations are crucial here, as they simulate the variance of the data and force the model to focus on relevant features.
Definition:
“Self-supervised learning with data augmentation is a learning paradigm where a model learns robust representations through artificially generated variations of the input data by producing similar representations for different augmentations of the same original.”
Source:
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A Simple Framework for Contrastive Learning of Visual Representations. Proceedings of the 37th International Conference on Machine Learning (ICML).