Sequence to Sequence Learning with Neural Networks

Overview

This paper helped popularize sequence-to-sequence learning as a general way to map one variable-length sequence into another. The paper’s main example is English-to-French machine translation, but the same abstraction later influenced summarization, dialogue, question answering, speech recognition, and code generation.

Before this work, machine translation was largely dominated by statistical machine translation. Those systems were strong, but they were complicated pipelines. This paper showed that a large deep neural model could directly compete with those systems.

The story behind the paper

In the early 2010s, deep learning had already achieved major results in speech and vision, but full sentence generation in NLP was still difficult. Translation was especially challenging because the input and output sentences can have different lengths and different word orders.

The authors — Ilya Sutskever, Oriol Vinyals, and Quoc V. Le — proposed a clean alternative to hand-engineered translation pipelines. Their model uses an encoder-decoder architecture built from LSTMs.

This was not the first neural translation-related work, but it was one of the clearest demonstrations that a pure neural system could perform large-scale translation directly.

Abstract and introduction

The paper argues that a domain-independent method for mapping sequences to sequences would be useful. Translation is the main task in the paper, but the framing is much broader.

The authors evaluate on the WMT14 English-to-French dataset and report results using BLEU score. Their best direct neural system reaches 34.81 BLEU, compared with 33.30 BLEU for the phrase-based SMT baseline.

The abstract also introduces one of the paper’s most surprising tricks: reversing the source sentence. This simple preprocessing step makes the optimization problem much easier.

The model

A vanilla RNN can process sequences, but it struggles with long-range dependencies. This paper uses LSTMs because translation often requires information from earlier words to influence much later output words.

The model uses one LSTM to read the input sentence and produce a fixed-dimensional vector. A second LSTM then generates the target sentence one word at a time. The output process stops when the model produces an end-of-sentence token.

The model has three important implementation choices: two separate LSTMs, multiple LSTM layers, and reversed source sentences.

The source reversal trick changes training from:

into:

This is not a linguistic claim. It is an optimization trick. It makes some source-target dependencies shorter, which helps gradient descent learn the mapping.

Mathematical formulation

The model estimates a conditional probability:

Here, x1, ..., xT is the input sequence, and y1, ..., yT′ is the output sequence. Their lengths do not need to match.

The decoder generates the target sentence one token at a time, so the probability can be decomposed using the chain rule:

Intuitively, the model predicts the first target word, then predicts the second target word given the first, then predicts the third given the previous words, and so on.

During inference, the model tries to find the most likely output sentence. Since trying every possible sentence is impossible, the paper uses beam search.

Experiments

The training set contains roughly 12 million sentence pairs. The model uses a fixed vocabulary, so out-of-vocabulary words become UNK tokens. The model is large for its time: four LSTM layers, 1000 cells per layer, 1000-dimensional embeddings, and hundreds of millions of parameters.

Training recurrent networks can be unstable, so the authors use gradient clipping. They also batch sentences of similar length together to reduce wasted computation.

The paper tests two uses of the LSTM. First, direct translation: the model generates the French translation itself. Second, rescoring: the model re-ranks the 1000-best translations produced by an SMT system.

Reversing the source sentence also reduces perplexity and significantly improves BLEU. This shows that problem encoding can be just as important as model architecture.

Why previous methods were not enough

Traditional SMT systems were strong, but they were highly engineered. They required many components and assumptions: phrase tables, word alignments, language models, and reordering systems. They were not simple end-to-end neural learners.

Vanilla RNNs could process sequences, but they struggled with long-term dependencies and were hard to use for tasks where input and output lengths differ. LSTM helped with this problem, but the paper still needed source reversal and careful training.

Earlier neural approaches often used neural networks as helper modules inside SMT systems. This paper’s major step was to show that a deep LSTM encoder-decoder could directly translate at large scale.

The paper still has an important limitation: the fixed-vector bottleneck. The encoder compresses the whole source sentence into one vector. Later attention mechanisms would address this limitation.

Impact and historical position

The paper’s historical importance is not just that it beat an SMT baseline. Its deeper contribution is that it helped establish a general modeling language for NLP generation tasks:

Today, this model is no longer state of the art. It has a fixed-vector bottleneck, limited vocabulary, recurrent computation, and high training cost. But historically, it helped shift NLP from hand-engineered pipelines toward end-to-end sequence learning.