Routing by Agreement is an iterative algorithm in Capsule Networks that adjusts routing coefficients \( c_{ij} \) between lower capsules \( i \) and upper capsules \( j \) based on the agreement of their activations.
The prediction of a lower capsule for an upper one is \( \hat{u}_{j|i} = W_{ij} u_i \), where \( W_{ij} \) is a weight matrix and \( u_i \) is the activation vector of the lower capsule.
The summation \( s_j = \sum_i c_{ij} \hat{u}_{j|i} \) is normalized by a nonlinear squash function to \( v_j = squash(s_j) \).
Routing coefficients are defined by softmax normalization of the routing logits \( b_{ij} \):
\[ c_{ij} = rac{\exp(b_{ij})}{\sum_k \exp(b_{ik})} \]
The routing logits are iteratively updated based on the agreement between \( \hat{u}_{j|i} \) and \( v_j \):
\[ b_{ij} \leftarrow b_{ij} + \hat{u}_{j|i} \cdot v_j \]
This adjustment promotes strengthening connections where the predictions of the lower capsules agree with the output of the upper capsules, effectively modeling hierarchical relationships between features.
Definition:
“Routing by Agreement is an iterative routing mechanism in Capsule Networks that adjusts connection strengths between capsule layers based on the agreement of their activations to model hierarchical feature relationships.”
Source:
Sabour, S., Frosst, N., & Hinton, G. E. (2017). Dynamic Routing Between Capsules. Advances in Neural Information Processing Systems (NeurIPS).