The attention mechanism is a neural architectural principle that allows relevant contextual information to be adaptively weighted in order to improve the modeling of dependencies in data.
Formally, attention computes the output for a query q, a set of keys k_i, and corresponding values v_i as a weighted sum of the values:
A(q, K, V) = Σ_i α_i v_i, where α_i = softmax(e_i) and e_i = f(q, k_i) represents a compatibility function, e.g., the dot product.
The mechanism allows flexible focusing on relevant parts of the input, which brings decisive advantages especially in sequence processing.
Transformer models (Vaswani et al., 2017) use multi-head attention to capture different aspects of the data in parallel and effectively integrate contextual information.
Definition:
The attention mechanism is a method in neural networks that enables adaptive context focusing by computing weighted combinations of input elements.
Source:
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is All You Need. In Advances in Neural Information Processing Systems (NeurIPS).