Why Does Transformer Need Positional Encoding?

The Transformer's core architecture is Self-attention (self-attention mechanism). In original Self-attention computation, regardless of input token order, output vectors are identical. This creates serious language comprehension problems: "you hit me" and "I hit you" become indistinguishable to the model. Therefore, we must introduce "position information" so models know each token's exact location.

From Absolute to Relative: Positional Encoding Evolution¶

1. Absolute Positional Embedding¶

The earliest approach designed unique vectors for each position (P₀, P₁…), directly adding them to token embeddings. The most famous design is Sinusoidal encoding, combining sine/cosine functions at different frequencies. The clever design lets models potentially understand "relative distance" through linear transformations.

2. ALiBi (Attention with Linear Biases)¶

Research found relative positions matter more than absolute ones. ALiBi takes a "crude" approach: no extra position vectors. Instead, during Attention score computation, it directly subtracts a constant bias based on token distance. Farther tokens get penalized more—letting models handle sequences longer than training data exceptionally well.

3. RoPE (Rotary Positional Embedding)¶

This is what mainstream models like Llama and Qwen use. RoPE's core concept is "rotation": it injects position info by rotating Query (Q) and Key (K) vectors. Its math guarantees Attention scores depend only on relative distance between Q and K. Fully compatible with KV Cache and acceleration (Flash Attention)—key reasons for popularity.

Extending Model Vision: Long-Text Expansion Techniques¶

When models process sequences exceeding training length (Train Short, Test Long), RoPE faces challenges. Several expansion strategies:

• Position Interpolation: Shrink long sequence position numbers (divide by 2) fitting within seen ranges.

• NTK-aware Scaling: Frequency-based adjustment—high frequencies (fast-spinning hands) unchanged; low frequencies compressed.

• YaRN and Dynamic Scaling: Further optimizing different frequency scaling, or dynamically adjusting at inference based on input length—handling impressive 2M token lengths.

Soul Question: Do We Really Need Positional Encoding?¶

Surprisingly, multilayer Self-attention inherently contains position info. Research shows models without positional encoding (NoPE) sometimes better extrapolate. Yet explicit encoding significantly speeds training and reduces loss. Some research proposes DropPos: use encoding during training, then remove it near the end. This "burn bridges" strategy frees models from positional encoding constraints, achieving stronger long-text abilities.