Neural Machine Translation by Jointly Learning to Align and Translate

Overview

This paper is one of the key early works in neural machine translation. It starts from the standard RNN encoder-decoder setup and asks a very practical question: what happens when the input sentence is too long to fit comfortably into a single fixed-length vector?

The answer is what we now call Bahdanau attention. At every decoding step, the model computes a soft distribution over source words, forms a context vector from that distribution, and uses it to generate the next target word. In the paper's language, the model learns to align and translate jointly.

The story behind the paper

Around 2014, sequence-to-sequence models made a bold promise: translation could be learned end-to-end. An encoder would read an English sentence, and a decoder would generate a French sentence. This was much cleaner than traditional phrase-based pipelines.

But the clean design had a suspiciously narrow bridge in the middle. The encoder had to compress the entire source sentence into one vector c. For short sentences, this can work. For long sentences, it asks one vector to preserve every content word, syntactic dependency, reordering clue, and translation detail.

This also reconnects neural translation with a classic machine translation idea: word alignment. Earlier systems often treated alignment as a separate object. Bahdanau, Cho, and Bengio put alignment inside the neural network itself.

The problem: one fixed vector is too narrow

The background model represents translation as finding a target sentence y that maximizes p(y | x). A standard encoder-decoder first maps the source sentence into a context vector:

Then the decoder generates the target sentence token by token:

The important detail is that c is the same for every target position. The model uses the same compressed source representation when generating the first word, the tenth word, and the thirtieth word.

The model architecture

The encoder is a bidirectional RNN. For each source word xj, it concatenates a forward state and a backward state to form an annotation hj. This means each source position has access to both preceding and following words.

The decoder is still recurrent, but it no longer depends on one fixed source vector. It computes a new ci for every target position. This is why the model is called RNNsearch: each decoding step searches over the source annotations.

Model derivation: from a fixed vector to attention

The old encoder-decoder bottleneck

In the earlier RNN Encoder-Decoder architecture, the encoder reads the whole source sentence and compresses it into one fixed-length vector:

The decoder then uses this same vector c at every generation step:

This is elegant, but it creates a bottleneck. A short sentence and a long sentence must both be squeezed into one vector of the same size. When the input sentence becomes long, the decoder may lose access to details in the source sentence, especially details that appear late in the input.

Bahdanau et al. change exactly this point. Instead of forcing the encoder to produce one universal sentence vector, the encoder produces a sequence of source annotations:

The decoder then constructs a new context vector ci for each target position i. So the fixed-context form becomes a dynamic-context form:

The important new object is ci . It is not reused across the whole sentence. It is computed specifically for generating the i-th target word.

What is a source annotation?

A source annotation hj is not just the embedding of the word xj. It is a contextual representation produced by a bidirectional RNN. The forward RNN summarizes the words before and up to xj, while the backward RNN summarizes the words after and down to xj.

This means hj contains information about the whole source sentence, but it is most focused on the neighborhood around position j. This is why the decoder can treat the list h1, ..., hTx as a memory bank over the source sentence.

Step 1: score every source position

Before generating yi, the decoder has a previous hidden state si−1. This state summarizes what has already been generated on the target side:

The model compares this decoder state with every source annotation hj. The result is an alignment score eij:

This formula is often called additive attention. The decoder state and the source annotation are first projected by two learned matrices, Wa and Ua. Their transformed representations are then added together, passed through tanh, and finally projected to a scalar by vaT.

It is important not to read this as a simple word-to-word dictionary match. The score is not comparing a raw target word with a raw source word. It compares:

The score eij asks: given the current state of the translation, how relevant is this source position?

Step 2: normalize scores into attention weights

The scores ei1, ei2, ..., eiTx are raw numbers. To turn them into a distribution over source positions, the model applies softmax:

For a fixed target position i, the weights over all source positions sum to one:

This is the paper's soft alignment. A target word does not need to choose exactly one source word. It can distribute attention across several source positions.

Step 3: build the step-specific context vector

Once the model has the attention weights, it forms the current context vector by taking a weighted average of all source annotations:

This equation is the heart of the model. The context vector ci is an information packet assembled for the current target word. If one source position is highly relevant, its annotation receives a large weight. If several positions are jointly useful, the model can combine them.

For example, when translating:

When the decoder is about to generate signé, the weight on the English annotation for signed should become large. The resulting ci then carries the relevant source-side information into the decoder.

Step 4: update the decoder and predict the next word

After the context vector is built, the decoder updates its hidden state:

In the implementation, f is not a plain RNN update. It is a gated recurrent update, similar in spirit to a GRU. The update gate decides how much of the previous state to keep, and the reset gate decides how much previous information should be ignored when forming a candidate new state.

The exact gate equations are implementation details, but they matter conceptually because the attention context ci participates directly in the recurrent update. The decoder is not merely looking at the source after it updates; the source context helps determine the new decoder state itself.

Finally, the model predicts the next target word using the previous word, the current decoder state, and the current context:

The implementation uses a small neural output network with a maxout layer before the final vocabulary softmax. For understanding the paper, the central chain is:

Algorithm view

The model can be read as a retrieval procedure repeated at every decoding step. The source annotations act like a memory. The decoder state acts like a query. Attention computes which memory entries should be read.

Why this is called attention

The weights αij tell us where the decoder is looking when it generates a target word. If the target word is signé, the weight on signed should be high. If the target phrase changes word order, the attention peak can move non-monotonically. This is why the mechanism is more flexible than a left-to-right alignment pointer.

At the same time, this is not yet Transformer attention. Bahdanau attention is built on top of RNN encoder-decoder states. It does not need explicit positional encoding, because the bidirectional RNN already injects order information into hj. Transformer self-attention later removes recurrence, so it needs a separate way to represent position.

Computation and complexity

The price of attention is that the decoder no longer reads one fixed vector. At every target step, it must compare the current decoder state with every source annotation.

If the source sentence has length Tx and the target sentence has length Ty, then the model computes one alignment score for each pair (i, j):

This creates a rectangular attention matrix:

Each row corresponds to one target word. Each column corresponds to one source word. Reading across a row tells us which source positions the model attends to when generating that target word.

Cost of additive scoring

The alignment score is not just a dot product. It uses a small feedforward network:

If the alignment hidden size is da, then computing scores for all target-source pairs costs roughly:

The paper notes that Uahj does not depend on the target step i, so it can be precomputed for all source positions. This reduces repeated work during decoding, but the model still needs to combine each source-side transformed annotation with each decoder state.

Cost of building context vectors

After the scores are normalized, the context vector is a weighted sum:

If each annotation has dimension dh, then building context vectors across the whole target sentence costs roughly:

Why the tradeoff is worth it

Compared with a fixed-context encoder-decoder, this is more expensive. The old model encodes the source once and then reuses one vector c. Bahdanau attention pays an additional matching cost at every target step.

For machine translation, this cost was acceptable because typical sentences are not extremely long. The benefit was large: the model no longer had to compress a long sentence perfectly into a single fixed-length vector.

Relation to Transformer complexity

It is useful to distinguish this mechanism from Transformer self-attention. Bahdanau attention in this paper is mainly cross-attention from target steps to source positions, so the interaction pattern is:

Transformer self-attention compares tokens within the same sequence. For a sequence of length T, the interaction pattern is:

The conceptual connection is clear: both mechanisms replace a single compressed representation with pairwise relevance computation. The architectural difference is also clear: Bahdanau attention still relies on RNNs for order and state, while Transformer makes attention the main computation and adds positional information separately.

Experiment settings

The paper evaluates on WMT 2014 English-to-French translation. The authors train on a selected subset of large parallel corpora and evaluate on news-test-2014. They compare the proposed RNNsearch model against the conventional RNN encoder-decoder and against Moses, a strong phrase-based statistical machine translation system.

Each neural model uses a vocabulary of 30,000 frequent words per language. Words outside the shortlist become UNK tokens. This matters because the paper's model can be strong at sentence modeling while still losing BLEU when rare words appear.

Results and interpretation

The pattern is clear. RNNsearch beats RNNencdec under both length limits. RNNsearch-30 even beats RNNencdec-50, which suggests that the gain is not merely from seeing longer training sentences; the architecture itself is better.

The No UNK column is especially revealing. RNNsearch-50* reaches 36.15 BLEU on sentences without unknown words, slightly above Moses at 35.63. On the full test set, Moses is still stronger, mostly because rare-word handling remains a weakness of the neural model.

The length-based analysis in the paper supports the original hypothesis: the conventional encoder-decoder deteriorates sharply as sentence length grows, while RNNsearch is much more robust. This is exactly what we would expect if the fixed vector c were the bottleneck.

Impact and what is outdated today

The original model is not today's state of the art. It is still built on RNNs, it is hard to parallelize compared with Transformers, and it uses a word-level shortlist with UNK tokens. Modern systems usually use Transformer architectures and subword tokenization.

But the idea did not become outdated. It became common sense. The paper's durable contribution is the principle of dynamic information access: a model should retrieve the relevant parts of a sequence when it needs them, rather than relying on a single compressed representation.

This is why the paper is best understood as a bridge to Transformer attention. Bahdanau attention uses RNNs to carry order information and attention to read from the source. Transformers later remove recurrence and make attention the main computation, adding positional information explicitly because self-attention alone does not know token order.