Transformers are architectures for sequential data based on the self-attention mechanism that enable parallel processing.
The basic idea is the calculation of attention weights that determine how much one input element considers other elements.
Formally, attention is computed as a weighted sum of value vectors \( V \), where the weights come from the softmax function over the dot products of query and key vectors:
\[ \mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left( rac{QK^T}{\sqrt{d_k}} ight) V \]
Here, \( Q, K, V \) are the matrices for queries, keys, and values, and \( d_k \) is the dimension of the key vectors.
The transformer architecture consists of an encoder and decoder, each built from multiple layers with multi-head attention and feed-forward networks.
This structure enables learning complex dependencies in sequences and has revolutionized natural language processing.
Definition:
“Transformers are neural network architectures that enable parallel processing of sequences via self-attention and thus efficiently model long-range dependencies.”
Source:
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is All You Need. Advances in Neural Information Processing Systems (NeurIPS).