Attention Is All You Need

Overview

Attention Is All You Need introduces the Transformer, a model for sequence transduction that removes recurrence and convolution from the main architecture. The paper evaluates this idea on machine translation, but its impact reaches far beyond translation: the same architectural principle later becomes the foundation of BERT, GPT-style models, many vision models, and the modern large language model ecosystem.

The title is intentionally bold, but it should be read carefully. The Transformer is not made of attention alone. It still relies on embeddings, positional encodings, feed-forward networks, residual connections, layer normalization, dropout, and a final softmax layer. What the title really argues is more specific: for modeling relationships between positions in a sequence, attention can replace the recurrent and convolutional machinery that dominated earlier neural sequence models.

Why this paper mattered

Before the Transformer, strong translation systems usually depended on RNNs, especially LSTMs and GRUs. These models had a natural way to process order: read token 1, then token 2, then token 3. That was elegant, and it matched our intuition about language as a sequence. But it also created a computational bottleneck. The hidden state at position t could not be fully computed until position t−1 was done, which made long sequences harder to parallelize during training.

There was already a counter-movement. Models such as Extended Neural GPU, ByteNet, and ConvS2S tried to reduce sequential computation with convolutional sequence models. They replaced the step-by-step recurrence of RNNs with convolutional computation that could be applied to many positions at once.

The key difference was how far information could travel in one layer. ConvS2S used stacked convolutions: fast and parallel, but still local, so distant words needed multiple layers to influence each other. ByteNet used dilated convolutions to expand the receptive field more quickly, reducing the path between faraway positions to a logarithmic scale. Extended Neural GPU belonged to a related line of work that treated sequence processing more like repeated updates over a parallel memory-like grid. These models were important because they showed that sequence modeling did not have to be strictly recurrent. But they still did not make every token globally visible to every other token in a single step.

The Transformer asks a more radical question: if attention already lets a decoder look directly at any source position, why not let every token look directly at every other token? That is the step from attention as an alignment tool to self-attention as the core architecture.

The bottleneck it attacked

The central enemy is not simply that “RNNs are old.” The real problem is sequential dependence. In an RNN, information moves through a chain: the representation at one position depends on the representation before it. If token 1 needs to influence token 30, the signal must pass through many intermediate hidden states. This makes long-distance dependencies harder to learn and prevents the model from fully parallelizing computation across positions.

The architecture as a whole

The Transformer keeps the familiar encoder-decoder shape of earlier translation systems. The encoder reads the source sentence and builds contextual representations for every source token. The decoder generates the target sentence one token at a time. What changes is not the high-level translation workflow, but the internal mechanism used inside each block.

“Six layers” means six blocks with the same structure, not one block reused with shared weights. Each layer has its own parameters. The encoder stack repeats one kind of block six times; the decoder stack repeats a slightly larger block six times. This stacked design lets the model refine token representations gradually, from surface-level word information toward more task-specific contextual information.

Around each sublayer, the model applies a residual connection followed by layer normalization. This is written as LayerNorm(x + Sublayer(x)). These details are not decorative: residual connections preserve information and improve gradient flow, while layer normalization keeps the activations stable as the model becomes deeper.

Attention, QKV, and square-root scaling

The most useful way to understand attention is as soft retrieval. A token does not passively receive a hidden state from its neighbor. It asks a question, compares that question to candidate keys, and retrieves a weighted mixture of values.

The important point is that Q, K, and V are learned projections. Given token representations X, the model forms Q = XWQ, K = XWK, and V = XWV. The same token can therefore play three roles: it can ask for information, advertise itself to others, and provide content when selected.

The term QKT creates an n × n matrix of compatibility scores. Row i, column j means: how much should token i attend to token j? Softmax turns each row into attention weights, and multiplying by V creates weighted averages of value vectors.

Why divide by √d_k?

Suppose the components of q and k are independent, with mean 0 and variance 1. Their dot product is:

Since there are dk terms, the variance of the sum grows roughly like dk. In other words, larger key/query dimensions tend to produce larger and more extreme dot-product scores.

The problem shows up inside softmax. If the scores are moderately sized, softmax still leaves room for several positions to matter. For example:

But if the same pattern is scaled up, the distribution becomes much sharper:

Now one position receives almost all the probability, while gradients for the other positions become tiny. Dividing by √dk pulls the standard deviation of the dot products back toward a stable scale, keeping softmax from becoming too saturated. This is the “scaled” part of scaled dot-product attention.

Dot-product attention vs. additive attention

Dot-product attention scores compatibility with a simple vector dot product: score(q, k) = q · k. In matrix form, this becomes QKT, which is why it is so efficient: all query-key comparisons can be computed as one large, highly optimized matrix multiplication.

Additive attention, associated with Bahdanau attention, uses a learned scoring network instead of a direct dot product. A common way to write it is score(q, k) = vT tanh(Wqq + Wkk). In words, the query and key are first transformed, combined, passed through a nonlinearity, and then scored by another learned vector.

Additive attention can be expressive, and it was very important in earlier neural machine translation. The Transformer uses dot-product attention because it is much faster and more memory-efficient in practice, especially when implemented with batched matrix multiplication. The scaling factor √dk helps dot-product attention avoid the large-score problem that appears when the key/query dimension grows.

Multi-head attention

A single attention distribution gives the model one way to look at the sentence. But language contains many relationships at once: pronouns, modifiers, syntax, local context, long-range dependencies, and translation alignment. The multi-head design lets the model learn several views in parallel.

In the base model, dmodel = 512, h = 8, and each head uses dk = dv = 64. Eight heads of 64 dimensions concatenate back to 512 dimensions. This keeps the total cost similar to one full-width head while giving the model multiple learned subspaces.

Positional encoding

Self-attention by itself is permutation-friendly: without extra information, the model can see that a set of tokens exists, but not their order. That is why the model adds positional encoding to token embeddings.

Each position gets a dmodel-dimensional vector. Even dimensions use sine; odd dimensions use cosine. The frequencies differ by dimension. A single sine curve does oscillate between −1 and 1, but the whole position vector is a multi-frequency fingerprint. One dimension alone is not the point; the combination is.

Why not just use position numbers like 1, 2, 3? A raw number grows without bound and gives the network a poor representation of local and relative distance. Sinusoidal encodings stay numerically stable and create structured relationships between positions.

What “relative position is easier to learn” means

Many language patterns depend on offsets: previous word, next word, three tokens away, far away. The paper's sinusoidal choice makes a fixed offset k mathematically regular. Through trigonometric identities, PE(pos + k) can be represented as a linear function of PE(pos) for a fixed k.

Here, pos plays the role of the current position, and k is the offset. More precisely, each sinusoidal dimension uses a scaled position such as pos / wavelength, so the actual expression is closer to sin((pos + k) / wavelength). But the same idea still holds: for a fixed offset k, the representation at pos + k can be written as a predictable combination of the sine and cosine values at pos under the same frequency.

This is why sinusoidal encoding is useful for relative distance. The model is not merely given isolated labels like “position 7” and “position 10.” It receives position vectors whose relationships contain regular patterns. That makes it easier for attention layers to learn ideas like “look one token back,” “look three tokens ahead,” or “prefer nearby tokens,” even though those rules are not hard-coded.

This does not mean the model is handed an alignment table. It means the encoding gives the model a convenient structure for learning “distance-like” relationships. In translation, English and French words often move to different positions. Positional encoding only tells each side its own internal order; the actual cross-language correspondence is learned by encoder-decoder attention.

Why self-attention

The case for self-attention rests on three questions. How much computation does one layer need? How many steps must be executed sequentially? And how short is the path between two distant positions? A good sequence layer should be parallelizable and should let faraway tokens exchange information through short paths.

Where the complexity numbers come from

When is self-attention cheaper than recurrence?

The key comparison is self-attention versus recurrence. Self-attention costs O(n²d); recurrence costs O(nd²). Ignoring constant factors, self-attention is cheaper when:

In sentence-level translation, the sequence length n is often much smaller than the representation dimension d, such as dmodel = 512. This is why full self-attention can be attractive despite the quadratic term in sequence length.

The deeper advantage is not just raw complexity. Self-attention gives every pair of positions a direct route. That short path is especially useful for long-distance dependencies: a word near the beginning of a sentence can influence a word near the end without passing through a long chain of recurrent states or many layers of local convolution.

Training recipe

The paper's result is not architecture alone. It also depends on a careful training setup: subword tokenization, large batches, Adam, learning-rate warmup, dropout, label smoothing, checkpoint averaging, and beam search.

Learning rate schedule

The model uses Adam plus warmup. Early learning rate increases linearly for 4000 steps, then decays roughly like the inverse square root of the step number. Warmup prevents unstable early jumps; later decay makes updates more precise.

Dropout is applied after sublayer outputs and after the sum of embeddings and positional encodings. Label smoothing makes targets less overconfident. The paper notes a subtle trade-off: label smoothing can hurt perplexity because the model is less certain, while improving accuracy and BLEU.

For evaluation, the authors use checkpoint averaging: base models average the last 5 checkpoints, and big models average the last 20. They decode with beam search using beam size 4 and length penalty 0.6.

Results and comparisons

The headline result is simple: the Transformer is both strong and efficient. On WMT 2014 English-German, Transformer big reaches 28.4 BLEU, more than 2 BLEU above the previous best reported systems, including ensembles. On English-French, it reaches 41.8 BLEU as a single model.

The important comparison is not only “which model has the highest BLEU?” but also “how much computation did it take to get there?” Several earlier systems achieved strong translation quality by using recurrent models, convolutional models, mixture-of-experts layers, reinforcement-learning refinements, or ensembles. The Transformer competes with all of them while using substantially less training cost than many previous high-performing systems.

What the baselines are doing

The baselines are not random older models; they represent the strongest competing directions in neural machine translation at the time. Some are recurrent encoder-decoder systems with attention, some are convolutional sequence models, some use very large sparse computation, and some combine multiple models into an ensemble. This matters because the Transformer is not being compared only against weak RNNs. It is being compared against several serious attempts to make translation models more accurate or more efficient.

Read these baselines as a map of the field before the Transformer. GNMT represents the mature recurrent translation pipeline: deep LSTMs, attention, and production-oriented engineering. ConvS2S and ByteNet represent the convolutional push toward better parallelism. Deep-Att + PosUnk represents an attention-heavy recurrent translation line with explicit handling of rare or unknown words. MoE explores scaling model capacity through sparsely activated expert networks. The ensemble rows show what happens when several trained models are combined at inference time, usually improving quality but increasing cost and complexity.

What the model variation experiments teach

The model variation results tell the architectural story behind the headline score. A single attention head is worse than multi-head attention, which supports the idea that different heads learn different relational views of the sentence. But too many heads can also hurt, because if the total model dimension is fixed, each head becomes too narrow to represent rich query-key compatibility.

Reducing the key/query dimension hurts quality, suggesting that deciding whether a query and a key match is not a trivial operation. Larger models generally perform better, dropout is important for preventing overfitting, and learned positional embeddings perform almost the same as sinusoidal encodings on the tested translation task. In other words, the paper’s best result is supported by a set of design choices rather than by one isolated trick.

Appendix visualizations

The appendix visualizes encoder self-attention heads. These figures are not strict proof that attention equals linguistic explanation, but they show why the model felt interpretable: some heads visibly focus on meaningful syntactic or semantic relationships.

Impact and afterlife

The original Transformer is an encoder-decoder translation model. Modern LLMs usually use only part of that template. BERT is encoder-only and emphasizes understanding. GPT is decoder-only and emphasizes next-token generation. ChatGPT is not simply “the Transformer paper turned into a product”; it is a later stack of scale, pretraining, instruction tuning, feedback learning, safety work, and interface design.

Why was it called “Transformer”? At the paper level, the name fits the sequence-transduction framing: the model transforms one sequence into another. It also names the architectural transformation happening inside the model: representations are repeatedly transformed by attention and feed-forward blocks rather than updated by recurrence. The name became broader than the original machine-translation setting because the same block turned out to scale across many modalities and tasks.

The Google-versus-OpenAI question is best answered carefully. Google created the architecture and later used Transformer-based models in major products. OpenAI, however, pushed the decoder-only generative line into a public conversational interface. Architecture was necessary, but not sufficient; product timing and alignment strategy mattered too.