Transformer Paper Review: Attention is All You Need

Introduction

Google’s 2017 paper “Attention is All You Need” revolutionized the NLP field. This post analyzes the core ideas and structure of the Transformer.

Limitations of Previous Approaches

Problems with RNN/LSTM

• Sequential Processing: Cannot parallelize
• Long-range Dependencies: Gradient vanishing issues
• Training Speed: Very slow for long sequences

Core Ideas of Transformer

Self-Attention Mechanism

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

def scaled_dot_product_attention(Q, K, V, mask=None):
    """
    Q: Query matrix
    K: Key matrix
    V: Value matrix
    """
    d_k = K.shape[-1]
    scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
    
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
    
    attention_weights = F.softmax(scores, dim=-1)
    output = torch.matmul(attention_weights, V)
    
    return output, attention_weights

Multi-Head Attention

Uses multiple attention heads in parallel to extract information from different perspectives.

Architecture Structure

1. Encoder: Processes input sequence
2. Decoder: Generates output sequence
3. Positional Encoding: Adds position information

Why Revolutionary?

• Parallel Processing: Maximizes GPU efficiency
• Long-range Dependencies: Direct connections
• Scalability: Foundation for BERT, GPT, etc.

Practical Application Tips

Memory Optimization

When training large models, attention memory usage scales quadratically with sequence length. Use gradient checkpointing and mixed precision.

Training Stabilization

Warm-up learning rate schedule and layer normalization are essential.

Conclusion

Transformer is not just a new architecture—it has become the foundation for various fields beyond NLP, including computer vision (ViT) and multimodal (CLIP).

References

• Original Paper
• The Illustrated Transformer