Understanding Attention Mechanism

Intuitive Understanding

Attention mimics human attention — when we look at a painting, we don’t focus equally on every pixel but concentrate on key areas.

Scaled Dot-Product Attention

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

Why divide by $\sqrt{d_k}$? When $d_k$ is large, dot products become very large, causing softmax gradients to vanish. See Transformer Architecture Explained for the full implementation.