📄 A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation

Summary

Transformers often send a large share of attention to the first token regardless of its content. This "attention sink" hurts context use, quantization, streaming and interpretability. The authors trace how the sink is produced inside the 124M-parameter GPT-2, which uses learned query biases and absolute positional embeddings.

They combine structural analysis of attention scores with ten causal interventions, using inputs from natural language, math and code. The sink comes from three interacting parts:

• The query bias b_Q. It contributes a score term that favours a given target position for every source token alike.
• The first-layer MLP. It turns the first positional embedding into an "effective positional encoding" (EPE) with massive activations on a few coordinates.
• The key projection W_k. Its structure lines up those coordinates with the bias direction.

Breaking any of the three weakens, removes or relocates the sink. Other architectures still show sinks while lacking each of these parts. The authors therefore argue that sinks arise through different circuits in different models. As a result, mitigations may need to be tailored to each architecture, and pre-training fixes may not transfer.

Key points

• Data: 300 inputs, 100 each from SST-2, GSM8K and HumanEval. All were cut to exactly 40 tokens, and per-head analyses cover layers 4–11 of 12.
• EPE fidelity: EPE (first-layer MLP output plus the positional vector) closely matches the actual positional signal. Its cosine similarity exceeds 0.82 at every position and 0.99 at position 1.
• Causal tests:
  • Nullifying b_Q sharply reduces the sink.
  • Zeroing the three W_k columns at EPE₁'s massive-activation coordinates also reduces it, while zeroing three random columns does almost nothing.
• Ruled-out explanations: The sink survives when the BOS embedding is zeroed and when only raw positional embeddings are swapped. This points to the MLP-processed signal rather than token identity.
• Code: Released at github.com/YuvMilo/MechanisticAccountofSinks.