Elamur Research⌕
☰
home / concepts / Transformer and attention

⚡ Transformer and attention

Concept · 30 key papers · all concepts

The Transformer is a neural network architecture that represents a sequence by letting each element gather information from all others through learned, weighted comparisons called attention. It replaces recurrence and convolution with stacked attention and feed-forward layers.

Also: self-attention network, scaled dot-product attention, multi-head attention (MHA), encoder–decoder Transformer

What it is

Attention is a mechanism that lets a network decide, for each output it produces, which parts of its input matter most. It computes a set of weights over input positions and uses them to form a weighted average of their representations. The weights are computed from the data, so they change from one input to the next.

The Transformer is the architecture built almost entirely from this operation. A sequence of tokens becomes a sequence of vectors. Each layer updates every vector in two steps:

Residual connections and layer normalization surround both steps. The original design dispensed with recurrent and convolutional components altogether (Vaswani et al. 2017). During training the whole target sequence is known in advance, so no step has to wait for the previous one, and all positions can be processed in parallel. Generating output at inference still proceeds one token at a time. This training-time parallelism is a major practical advantage over recurrent networks.

The key idea

The core operation is scaled dot-product attention:

Attention(Q, K, V) = softmax( Q Kᵀ / √d_k ) V

Q, K and V are matrices of queries, keys and values. Each is obtained by multiplying the input vectors by its own learned projection matrix. The three play different roles:

Kᵀ is the transpose of K, so Q Kᵀ holds the dot product of every query with every key. d_k is the dimension of the keys. Dividing by √d_k stops the dot products from growing with dimension and saturating the softmax. The softmax, applied row by row, turns scores into non-negative weights that sum to one. Multiplying by V gives each position a weighted mixture of values (Vaswani et al. 2017).

When Q, K and V all come from the same sequence, the operation is called self-attention. When the queries come from one sequence and the keys and values from another, it is called cross-attention.

Multi-head attention runs h such operations in parallel, each with its own projections into a smaller subspace. The h outputs are concatenated and projected back:

MultiHead(X) = Concat(head_1, …, head_h) W_O,   head_i = Attention(X W_Q,i , X W_K,i , X W_V,i)

Here X is the layer input, W_Q,i, W_K,i and W_V,i are the projections for head i, and W_O is the output projection. Some heads specialize in distinct relations, such as attending to an adjacent token or to a syntactic dependent (Voita et al. 2019).

Unmasked attention by itself is indifferent to word order: shuffling the inputs shuffles the outputs identically. Order must therefore be injected. The original model added fixed sinusoidal encodings of absolute position to the inputs (Vaswani et al. 2017). Later work used other schemes:

The full model has two halves (Vaswani et al. 2017):

Computing Q Kᵀ requires time and memory that grow with the square of sequence length. This cost recurs in every head of every layer (Tay et al. 2020).

History and founding papers

Attention in its modern sense arose in neural machine translation. Earlier encoder–decoder models squeezed a whole source sentence into one fixed-length vector. Bahdanau, Cho and Bengio instead let the decoder compute alignment weights over per-word annotations from a bidirectional RNN, and trained these weights jointly with the model. On WMT '14 English→French, their RNNsearch-50 scored 26.75 BLEU on the full test set, against 17.82 for the fixed-vector baseline. The gain was largest on long sentences (Bahdanau et al. 2014).

Luong, Pham and Manning then made two separate contributions (Luong et al. 2015):

The idea spread quickly to other tasks:

In 2017 Vaswani and colleagues removed the recurrent backbone entirely. They introduced scaled dot-product and multi-head attention within an encoder–decoder stack (Vaswani et al. 2017). Refinements followed soon after. Relative position representations raised Transformer-big BLEU on WMT 2014 English–German from 27.9 to 29.2 (Shaw et al. 2018).

How it is used today

The original encoder–decoder split into several families, distinguished by which half is kept and how it is pre-trained.

FamilyTypical trainingExamples
Encoder-onlymasked-token prediction, bidirectional contextBERT (Devlin et al. 2019), AraBERT (Antoun et al. 2020)
Decoder-onlynext-token predictionCTRL (Keskar et al. 2019), OPT (Zhang et al. 2022)
Encoder–decodertext-to-text transferT5 (Raffel et al. 2019), mT5 (Xue et al. 2021)

Decoder-only models generate text and can also be steered. CTRL, a 1.63-billion-parameter model, conditions every sequence on a control code derived from metadata (Keskar et al. 2019). Fine-tuned generators have been used to write new entries for commonsense knowledge graphs (Bosselut et al. 2019). An open-source library offers many of these architectures through one consistent interface, together with a hub for sharing pretrained weights (Wolf et al. 2020).

Beyond text, the architecture spread to several fields:

Other variants cut the cost of long inputs or of decoding. Longformer combines local and global attention patterns to handle long documents (Beltagy et al. 2020). Grouped-query attention shares key and value heads among groups of query heads to reduce memory traffic during decoding. On T5 XXL, eight-group GQA cut time per sample from 1.51 ms with standard multi-head attention to 0.28 ms. Its quality score barely moved, at 47.1 versus 47.2; the score averages summarization (ROUGE-1), translation (BLEU) and question answering (F1) (Ainslie et al. 2023).

Open questions

Cost remains the most visible problem. Dozens of efficient variants, nicknamed "X-formers", exist, yet none has displaced standard quadratic attention. Linear-attention variants often trail on benchmarks, and claimed efficiency does not always translate into faster wall-clock time (Tay et al. 2020).

Interpretability is unsettled. Many heads appear redundant: pruning 38 of 48 encoder heads in an English–Russian model cost only 0.15 BLEU (Voita et al. 2019). Raw attention weights are also a weak guide to which inputs matter. One subject–verb agreement study measured Spearman correlation with blank-out ablation importance over 2000 test samples. In the final layer, raw attention scored about 0.29, against 0.71 for attention rollout (Abnar & Zuidema 2020). Whether attention maps can count as explanations is still debated.

Inductive bias is another open issue. In speech, encoders built only from self-attention handled fine local acoustic patterns poorly. On speech and vision benchmarks, adding convolutions or local windows helped (Gulati et al. 2020; Liu et al. 2021). Whether this reflects a general weakness is unclear, and there is no principled rule yet for how to mix the two.

How best to encode position is also unresolved, especially for inputs longer than any seen in training (Shaw et al. 2018; Su et al. 2021).

Finally, the memory cost of storing keys and values during inference continues to shape new head designs (Ainslie et al. 2023).

Key papers

The works this page cites, the most cited first; citations as counted by OpenAlex plus the papers of this library that cite the work, read 2026-10-23.

More papers

Topics: 🧠 Language models & pretraining · 🔁 Machine translation · 🔬 Interpretability

Shelves: 💬 Natural language processing · 🤖 AI & machine learning

This page in Markdown → https://md.research.elamur.ai/free/en/concepts/transformer/