Self-attention is a mechanism where, for an input sequence X = (x_1, x_2, ..., x_n), a weighted sum of all elements is computed for each element x_i, with the weights determined by the compatibility of x_i with each x_j.
Formally, this is often realized through the Query (Q), Key (K), and Value (V) matrices, where the attention scores are calculated from the dot product of Q and K normalized via softmax:
Attention(Q,K,V) = softmax( (Q K^T) / sqrt(d_k) ) V
Here, d_k is the dimension of the key vectors, used for scaling.
This mechanism allows modeling contextual dependencies flexibly and in parallel, which is especially utilized in transformer architectures (Vaswani et al., 2017).
Definition:
“Self-attention is a mechanism in neural networks that models the contextual relationships of elements in an input sequence by computing weighted combinations of the elements, where each element attends to all other elements in the sequence.”
Source:
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is All You Need. In Advances in Neural Information Processing Systems (NeurIPS).