What is Transformer

Transformer was proposed by Google in 2017 and completely revolutionized NLP. It is based entirely on the attention mechanism, abandoning traditional RNN and CNN structures.

Self-Attention

The core idea of Self-Attention is to allow each token in a sequence to directly attend to all other tokens:

def self_attention(Q, K, V):
    scores = Q @ K.T / math.sqrt(d_k)
    attention_weights = softmax(scores)
    return attention_weights @ V

For a deeper mathematical analysis, see Understanding Attention Mechanism.