The Transformer architecture (Vaswani et al., 2017) is a sequence-based model that relies on a self-attention mechanism to process input sequences in parallel and globally model contextual dependencies.
Formally, a Transformer consists of multiple encoder and decoder layers, with each encoder layer containing a multi-head self-attention (MHSA) component and a positional feedforward network (FFN).
The self-attention for an input sequence \(X = (x_1, ..., x_n)\) computes weighted sums over all tokens for each token:
\[ \mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V \]
Here, queries \(Q\), keys \(K\), and values \(V\) are linear projections of \(X\), and \(d_k\) is the dimension of the key vectors.
Multi-head attention extends this by parallel attention heads that capture different aspects of the input.
Positional encodings are added to account for the order of the tokens, as self-attention itself is position-independent.
The Transformer architecture has revolutionized the NLP field and is the basis for models like BERT, GPT, and T5, which achieve state-of-the-art performance on numerous language tasks.
Definition:
“Transformers are neural network architectures based on self-attention to process sequences in parallel and efficiently model contextual dependencies.”
Source:
Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention Is All You Need. In Advances in Neural Information Processing Systems (NeurIPS).