Attention Is All You Need, Nine Years Later: An Audit of What Survived

Attention Is All You Need went up on arXiv on 12 June 2017, which makes it nine years and a day old today. I went back and read it properly this year rather than relying on the version of it I had absorbed from other people’s blog posts, and the thing that surprised me was not how much of it turned out to be right. It was how little of the actual architecture is still in use.

Almost every specific engineering choice in that paper has since been replaced. What survived is one equation, one block shape, and one argument. This is a component by component audit of which is which.

What the paper actually proposed

It is worth being precise about this, because the paper has been retconned into something larger than it was. It is a machine translation paper. It reports BLEU on WMT 2014 English-to-German and English-to-French. It contains no mention of foundation models, scaling laws, or in-context learning, because in June 2017 none of those were things. The claim in the abstract is modest and mostly about cost: the model is “superior in quality while being more parallelizable and requiring significantly less time to train”.

BASE

d_model 512, 6 layers, 8 heads
d_ff 2048, dropout 0.1
65M parameters
100k steps, 12 hours on 8 P100s

BIG

d_model 1024, 6 layers, 16 heads
d_ff 4096, dropout 0.3
213M parameters
300k steps, 3.5 days on 8 P100s

Two hundred and thirteen million parameters was the big model. Adam with a 4,000 step warmup, label smoothing at 0.1, sinusoidal positional encoding, layer normalisation after each sublayer, ReLU in the feed-forward network, an encoder stack and a decoder stack joined by cross-attention. Every one of those choices except the optimiser has since been changed.

The part that survived

One equation:

Attention(Q, K, V) = softmax(QK^T / √d_k) V

Including the scaling factor, which the paper justifies in a footnote: for large d_k the dot products grow large in magnitude, pushing the softmax into regions where its gradient vanishes. That footnote is still load bearing in every model shipping today.

Two other things survived, in spirit if not in detail:

Multi-head projection. Running several attention operations in parallel over lower-dimensional projections and concatenating them. Still universal, although how the heads share their keys and values has changed completely.
The block shape. An attention sublayer, then a position-wise feed-forward sublayer, with a residual connection around each. Nine years of research has not found a reason to change the skeleton.

That is the list. Everything below it is the audit.

Sinusoidal positions became RoPE

The paper adds fixed sinusoids to the input embeddings and explains the choice with a hypothesis: it “may allow the model to extrapolate to sequence lengths longer than the ones encountered during training”. In practice it did not, and absolute position added at the embedding turned out to be the wrong place to put the information.

Rotary position embedding (Su et al., 2021) rotates the query and key vectors by an angle proportional to their position, so the dot product between two of them depends on the offset between them rather than on where each one sits in the sequence. Position stops being something you add to the input and becomes a property of the attention operation itself. It is now the default in Llama, Gemma, Qwen and DeepSeek. Several recent models go further and drop positional encoding entirely in some layers, on the grounds that a causal mask already carries ordering information.

Post-LayerNorm became pre-RMSNorm

The 2017 block computes LayerNorm(x + Sublayer(x)), normalising after the residual. Xiong et al. (2020) showed that this placement is why the paper needs its learning rate warmup at all: with post-norm the gradients at initialisation are large enough that training diverges without one. Moving the normalisation inside the residual branch removes the requirement.

Separately, RMSNorm (Zhang and Sennrich, 2019) drops the mean subtraction and the learned bias and keeps only the scale, which costs less and works about as well. Current models combine the two, and several add another normalisation to the queries and keys inside attention.

This one changed how I read papers. The 4,000 step warmup in the transformer paper is not a general truth about training deep networks. It is a symptom of one design choice that has since been reversed. Carrying it into a pre-norm model because a famous paper did it is cargo cult.

The ReLU feed-forward became SwiGLU, then sometimes disappeared

The paper’s feed-forward network is two linear layers with a ReLU in between. Shazeer (2020) tested gated linear unit variants in the same slot and found SwiGLU consistently better for the same parameter budget, and it is now the standard.

The larger change is that in the biggest models the dense feed-forward layer is gone, replaced by a sparse mixture of experts that routes each token to a handful of specialists out of hundreds. DeepSeek-V3 has 671B total parameters and activates roughly 37B per token. The distinction between how big a model is and how much of it runs did not exist in 2017.

Multi-head attention became grouped-query attention

This is the change with the most interesting cause, because it has almost nothing to do with model quality.

In the original design every head has its own key and value projections. During training that is fine. During autoregressive decoding you cache the keys and values for every head at every position, and that cache has to be read from memory on every single generated token. The bottleneck is not arithmetic, it is memory bandwidth.

Shazeer (2019) proposed sharing one key/value head across all query heads, which shrinks the cache dramatically at some cost in quality. Grouped-query attention (Ainslie et al., 2023) is the compromise that stuck: query heads are divided into groups, and each group shares a key/value head. DeepSeek went a different way with multi-head latent attention, compressing keys and values into a low-rank latent before caching them.

None of these were motivated by accuracy. They exist because someone has to pay for inference. The 2017 paper had no reason to think about it: it was decoding sentences of about thirty tokens, once, in a research setting.

Encoder-decoder became decoder-only

Translation has a source and a target, so the paper has an encoder stack, a decoder stack and cross-attention between them. Once the task generalised to “predict the next token over everything”, the encoder stopped earning its parameters. Every current general purpose language model is decoder-only. This is the one deletion in the list rather than a replacement.

The complexity argument aged worst

The sentence in that paper that has aged least well is not in the abstract. It is in Table 1, which compares self-attention to recurrence and convolution and notes that self-attention is faster when the sequence length is smaller than the representation dimension. For machine translation that was comfortably true: sequences of about thirty tokens against a model dimension of 512.

That inequality inverted, and essentially the whole engineering history of the last four years is a response to it. FlashAttention (Dao et al., 2022) does not change the mathematics at all and never materialises the full attention matrix in high bandwidth memory, which is a pure memory-access win. Sliding window attention restricts most layers to a local neighbourhood. Hybrid stacks interleave linear attention layers with a small number of full attention layers.

The paper’s core argument, in the same table, is the one that held: self-attention gives a constant maximum path length between any two positions, and it has no sequential dependency across the sequence. That second property is the reason the architecture scaled. It was the first sequence model that could saturate a GPU.

The same block, then and now

The 2017 attention function, more or less verbatim from the paper:

def attention(q, k, v, mask=None):
    d_k = q.size(-1)
    scores = q @ k.transpose(-2, -1) / math.sqrt(d_k)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float("-inf"))
    return torch.softmax(scores, dim=-1) @ v

A block from a current decoder-only model, with the head reshaping elided:

def block(x, pos, w):
    h = rms_norm(x, w.attn_norm)          # pre-norm, RMS not Layer
    q, k, v = h @ w.wq, h @ w.wk, h @ w.wv  # k and v have fewer heads than q
    q, k = apply_rope(q, pos), apply_rope(k, pos)   # position enters here
    k, v = repeat_kv(k, n_rep), repeat_kv(v, n_rep) # grouped-query
    x = x + F.scaled_dot_product_attention(q, k, v, is_causal=True) @ w.wo
    return x + swiglu(rms_norm(x, w.ffn_norm), w)   # not a ReLU MLP

Every line differs from 2017 except the contents of scaled_dot_product_attention, which is the first snippet with a better memory access pattern.

A footnote about Noam Shazeer

Shazeer is the second author on Attention Is All You Need. He is also the sole author of the multi-query attention paper that began dismantling its attention block, the sole author of the GLU paper that replaced its feed-forward network, and the first author of the sparsely-gated mixture of experts paper, which predates the transformer paper by five months and describes the thing that eventually replaced the feed-forward layer outright.

The person who did the most to take the 2017 design apart is one of the people who wrote it. That is worth noticing if you are inclined to treat a famous paper as a finished object.

What I take from this

The durable contribution of a paper is usually not its architecture, it is the constraint the architecture removes. Here it was the sequential dependency in recurrence. Everything else was implementation, and implementation gets replaced.
Most of the replacements were driven by deployment economics, not accuracy. Grouped-query attention exists because of KV cache memory. FlashAttention exists because of memory bandwidth. Mixture of experts exists because of the cost of serving. If you read this literature only for benchmark numbers you miss what actually happened.
Hyperparameters from a famous paper are not laws. Warmup was a consequence of post-norm, not a property of transformers.
Read the tables. The claim that aged worst was a complexity bound in Table 1, and it set the research agenda for the following decade by being wrong about which term would dominate.

Nine years on, the title is still accurate and almost nothing under it is. For a paper in this field, that is an extremely good ratio.