NLP · BERT · 2018 / NAACL
BERT: Pre-training of Deep Bidirectional Transformers
A structured reading of the paper that turned the Transformer encoder into a general-purpose language-understanding engine.
Overview
Read BERT as a paper about how to pretrain understanding. The Transformer paper showed that attention could replace recurrence for sequence modeling. BERT asks a different question: if a Transformer encoder can let every token look at every other token, can we pretrain that encoder once and reuse it across many language-understanding tasks?
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding introduces , an encoder-only Transformer pretrained on ordinary text and then adapted to different . Its headline idea is simple: build a model whose token representations are deeply , then make the same pretrained model useful for classification, inference, question answering, and token labeling.
The important word is representation. BERT is not mainly a text generator. It is a model that turns tokens into contextual vectors: the representation of a word depends on the whole sentence, the sentence pair, and the task-specific fine-tuning signal. That is why the paper matters historically. It helped move NLP from hand-building a new architecture for every benchmark toward reusing a large pretrained model as the starting point for many tasks.
Why BERT appeared
BERT appears at a specific moment in NLP history. Researchers already knew that pretraining on large text corpora helped. Static word embeddings made downstream systems better; ELMo made word representations depend on context; OpenAI GPT showed that a Transformer could be pretrained and then fine-tuned. But the field still had an unresolved tension: the best fine-tuning recipe was , while many understanding tasks clearly require both left and right context.
A left-to-right is natural for generation: predict the next token from previous tokens. But language understanding often works differently. In question answering, a candidate answer token may need right-side context to determine its boundary. In natural language inference, one sentence must be interpreted against another. In named entity recognition, a token's type can depend on words before and after it.
The obvious solution, "let every token see both sides," creates a training problem. If the model is asked to predict a word while the original word is still visible in the input, it can cheat. BERT's central trick is to change the objective: hide some tokens first, then predict them. This is the . The architecture can be fully bidirectional because the target token itself is removed or corrupted.
ELMo had context but was mostly . GPT had but was left-to-right. BERT combines the useful parts: a Transformer fine-tuning framework, plus deeply bidirectional representations trained with a self-supervised objective.
The core idea
BERT's core idea is not merely "make a bigger Transformer." It is the alignment of three choices: an encoder architecture, a bidirectional pre-training objective, and a fine-tuning interface that can express many NLP tasks with small task heads.
The subtle point is that BERT's bidirectionality is legal because the target is hidden. A normal
language model cannot freely look left and right while predicting a visible word. BERT can, because
it first replaces selected words with [MASK], random words, or leaves them unchanged as
prediction targets. The model is trained to recover the original token, not to generate the next token
in a left-to-right stream.
This makes BERT especially natural for tasks where the answer is a representation, a class, or a span inside the input. It is less natural for open-ended generation than GPT-style models, but extremely powerful for language understanding, ranking, extraction, and classification.
Architecture
BERT's architecture is a stack of Transformer encoder blocks. The paper describes the model size with three symbols: L is the number of Transformer layers, H is the hidden size of each token representation, and A is the number of self-attention heads. So when the paper says BERTBASE has L=12, H=768, and A=12, it means every token is represented as a 768-dimensional vector through a 12-layer encoder stack with 12 attention heads per layer.
| Model | L | H | A | Parameters | Why it matters |
|---|---|---|---|---|---|
| BERTBASE | 12 | 768 | 12 | 110M | Chosen to be similar in size to OpenAI GPT for comparison. |
| BERTLARGE | 24 | 1024 | 16 | 340M | A larger version showing the benefit of scale under the same basic recipe. |
ELMo vs GPT vs BERT
The easiest way to understand BERT's architecture is to compare it with the two models it reacts to. ELMo showed that contextual representations help. GPT showed that pre-training plus fine-tuning can work well. BERT combines the useful parts of both, but replaces separate or left-to-right context with a deeply bidirectional encoder.
| Model | Pre-training structure | Directionality | Transfer style | Key point |
|---|---|---|---|---|
| ELMo | Two separate LSTMs | Left-to-right LM + right-to-left LM | Feature-based | Contextual embeddings are useful, but the two directions are trained separately and concatenated later. |
| OpenAI GPT | Transformer with causal LM | Left-to-right only | Fine-tuning | Fine-tuning is simple and powerful, but each token only sees previous tokens during pre-training. |
| BERT | Transformer encoder with MLM + NSP | Deep bidirectional | Fine-tuning | Every layer can combine left and right context because prediction targets are hidden or corrupted first. |
A normal left-to-right language model cannot simply look at both sides when predicting a token, because the original token would still be visible in the input. BERT solves this by changing the pre-training problem. It first hides or corrupts selected tokens, then asks the encoder to recover the original tokens. The architecture can attend bidirectionally because the answer token itself is no longer directly available.
One encoder, different task heads
The same pretrained BERT encoder can be reused for different downstream tasks. What changes is not the main encoder stack, but the input format and the small output layer. This is the practical reason BERT felt so powerful: many tasks could be reframed as "put the right sequence into BERT, then read the right output position."
This section is where BERT's "encoder-only" identity matters. GPT-style models are built for autoregressive generation, so their attention is causally masked. BERT removes that causal mask, uses a Transformer encoder, and relies on masked prediction to keep pre-training valid.
Input representation
BERT's input format is one of the paper's most practical contributions. It creates a single sequence format that can represent one sentence, two sentences, a question-passage pair, or a multiple-choice candidate. Every input starts with , and text spans are separated by .
input vector = token embedding + segment embedding + position embedding
These are added rather than concatenated because all three vectors have the same hidden dimension
H. For BERTBASE, each one is 768-dimensional, and their sum is still 768-dimensional.
The model learns the scale and use of each component during training. In plain language:
say what the token is,
say which text span it
belongs to, and say
where it appears.
| Surface token | WordPiece | Segment | Position | Role |
|---|---|---|---|---|
| [CLS] | [CLS] | A | 0 | Aggregate classification position. |
| my | my | A | 1 | Part of sentence A. |
| dog | dog | A | 2 | Part of sentence A. |
| is | is | A | 3 | Part of sentence A. |
| cute | cute | A | 4 | Part of sentence A. |
| [SEP] | [SEP] | A | 5 | Boundary after sentence A. |
| he | he | B | 6 | Part of sentence B. |
| likes | likes | B | 7 | Part of sentence B. |
| playing | play + ##ing | B | 8–9 | One word may become multiple WordPiece tokens. |
| [SEP] | [SEP] | B | 10 | Boundary after sentence B. |
Notice that tokenization can split a single word into multiple subword tokens. BERT predicts and represents WordPiece tokens, not always whole human-written words. This makes the vocabulary manageable while still covering rare or novel words.
Pre-training objectives
BERT is pretrained on unlabeled text, but that does not mean it has no training targets. The targets are created automatically from the text itself. This is why BERT's pre-training is better described as : ordinary text is converted into prediction problems before any human-labeled downstream dataset is used.
The pre-training to fine-tuning pipeline
BERT follows a two-stage recipe: first train a general encoder with self-supervised objectives, then copy the same pretrained parameters into different downstream models. The architecture is almost the same on both sides; the task-specific output layer changes.
Pre-training learns a general initialization from large text. Fine-tuning creates separate task-specific models from that same initialization. In the original BERT setup, MNLI, SQuAD, SWAG, and NER do not share one final multitask model; each task gets its own fine-tuned copy.
| Objective | Example input | Training target | Why it exists |
|---|---|---|---|
| Masked Language Model | my dog is [MASK] | Predict the original token, e.g. cute or hairy. | Allows bidirectional context while hiding the answer token. |
| Next Sentence Prediction | [CLS] sentence A [SEP] sentence B [SEP] | IsNext or NotNext. | Trains a sentence-pair representation useful for QA and NLI. |
Masked Language Model
The is BERT's answer to the bidirectionality problem. A standard language model predicts from one direction to avoid seeing the target token. BERT instead corrupts some input positions first. The encoder can then look left and right, because the original target token has been removed or changed.
BERT selects 15% of WordPiece tokens as prediction targets. Those selected targets are handled with the 80/10/10 masking strategy below. Notice that the model's target is always the original token, even when the input token is unchanged.
| Selected target case | Input shown to BERT | Prediction target | Purpose |
|---|---|---|---|
| 80% | my dog is [MASK] | hairy | The main cloze-style prediction case. |
| 10% | my dog is apple | hairy | Prevents the model from assuming every prediction target is visibly marked by [MASK]. |
| 10% | my dog is hairy | hairy | Reduces the mismatch between pre-training inputs and real fine-tuning inputs. |
Next Sentence Prediction
gives BERT a sentence-pair pre-training signal. For each pair, sentence B is the true next sentence 50% of the time and a random sentence 50% of the time. The final hidden state of is used to predict IsNext or NotNext.
NSP is most naturally connected to text-pair tasks. Question answering, natural language inference, and paraphrase detection all require the model to represent relationships between two pieces of text. Later models debated whether NSP was the best way to learn this signal, but in the original BERT paper it is part of the main recipe and is tested in the ablation studies.
Pre-training loss
BERT's pre-training loss has two parts: a token-level MLM loss and a sequence-level NSP loss. The paper uses the sum of these losses. Conceptually, the model is punished when it fails to reconstruct selected original tokens, and also when it fails to classify whether the second text segment is the real next segment.
BERT is trained with two losses at the same time: one for reconstructing selected tokens, and one for judging whether two text segments are consecutive.
LMLM is computed only over the selected 15% prediction positions.
LNSP is computed from the binary IsNext / NotNext classifier using
the [CLS] representation.
| Loss part | Where the signal comes from | Target | Role in pre-training |
|---|---|---|---|
| MLM loss | Only the selected 15% prediction positions contribute to this loss. | The original WordPiece token before masking, random replacement, or keeping unchanged. | Teaches BERT to reconstruct missing or corrupted tokens from bidirectional context. |
| NSP loss | One binary classification loss for the [CLS] representation. | IsNext or NotNext. | Teaches BERT a sentence-pair signal useful for question answering and natural language inference. |
| Total pre-training loss | The sum of MLM loss and NSP loss. | Token reconstruction plus sentence-pair classification. | Combines token-level contextual learning with sequence-level relationship learning. |
The MLM loss is not computed for every token in the sequence. It is computed only at the selected prediction positions, which are 15% of WordPiece tokens. For each selected position, BERT takes the final hidden vector at that position and predicts a distribution over the vocabulary. The correct class is the original token id. This is the same cross-entropy idea used in many classification problems, but here the classes are vocabulary items.
The NSP loss is a binary cross-entropy classification loss. The input pair is encoded by BERT, the
[CLS] vector is read as the sequence-pair representation, and the classifier predicts
whether the pair is IsNext or NotNext. The final pre-training update
combines this sentence-pair loss with the masked-token reconstruction loss.
MLM makes deep bidirectional pre-training possible. NSP adds a sentence-pair signal. Together they let BERT learn both local token meaning and broader text-pair relationships before any supervised downstream fine-tuning.
Fine-tuning
BERT's downstream recipe is deliberately plain. Start from the same pretrained checkpoint, add a small task-specific output layer, and train all parameters on the labeled dataset. This is the fine-tuning approach, not just feature extraction. Each downstream task gets a separate fine-tuned model, but they all begin from the same pretrained BERT.
| Task type | BERT input | Output | Example |
|---|---|---|---|
| Single-sentence classification | [CLS] sentence [SEP] | Class label from the [CLS] vector. | SST-2 sentiment: positive or negative. |
| Sentence-pair classification | [CLS] sentence A [SEP] sentence B [SEP] | Class label from the [CLS] vector. | MNLI: entailment, contradiction, or neutral. |
| Extractive question answering | [CLS] question [SEP] passage [SEP] | Start and end token positions. | SQuAD: select the answer span from the passage. |
| Token classification | [CLS] token sequence [SEP] | A label for each token. | NER: person, location, organization, or other. |
The most elegant case is extractive question answering. BERT reads the question and passage as one
packed sequence. Then the task head scores each passage token as a possible start and a possible end.
For SQuAD v2.0, where some questions have no answer, BERT treats the [CLS] position as
the no-answer span and compares that score against the best non-null span.
This is why BERT reduced the need for heavily engineered task architectures. Earlier systems often designed separate cross-attention modules for each task. BERT lets the self-attention inside the packed input perform much of that interaction automatically.
Tasks and benchmarks
The paper's benchmarks are not random. They cover sentence-level classification, sentence-pair reasoning, token-level extraction, and commonsense-style multiple choice. This breadth is the evidence behind the claim that BERT learns general-purpose language representations.
| Dataset / benchmark | Task | Input | Output | Example |
|---|---|---|---|---|
| MNLI — Multi-Genre Natural Language Inference | Natural language inference across multiple genres | Premise + hypothesis | Entailment / contradiction / neutral | Premise: “A man is playing guitar.” Hypothesis: “A person is making music.” → entailment. |
| QQP — Quora Question Pairs | Paraphrase detection for user questions | Two Quora questions | Duplicate or not | “How can I learn Python?” + “What is the best way to study Python?” → likely duplicate. |
| QNLI — Question Natural Language Inference | Question answering converted into sentence-pair classification | Question + candidate sentence | Contains answer or not | Question: “Where was Tesla born?” Sentence: “Tesla was born in Smiljan.” → contains answer. |
| SST-2 — Stanford Sentiment Treebank | Single-sentence sentiment classification | One movie-review sentence | Positive or negative | “A warm, funny, and beautifully acted film.” → positive. |
| CoLA — Corpus of Linguistic Acceptability | Grammar-sensitive acceptability judgment | One sentence | Acceptable or unacceptable | “The book is on the table.” → acceptable; “Book the table on is.” → unacceptable. |
| STS-B — Semantic Textual Similarity Benchmark | Graded semantic similarity | Two sentences | Similarity score | “A man is riding a bicycle.” + “A person is biking.” → high similarity. |
| MRPC — Microsoft Research Paraphrase Corpus | Paraphrase detection in news text | Two news sentences | Equivalent or not | “The company reported higher profits.” + “The firm said earnings increased.” → likely equivalent. |
| RTE — Recognizing Textual Entailment | Small-data textual entailment | Text + hypothesis | Entailment or not | Text: “The treaty was signed in Paris.” Hypothesis: “The agreement was signed in France.” → entailment. |
| SQuAD v1.1 — Stanford Question Answering Dataset | Extractive question answering | Question + passage | Answer span | Question: “Where was Tesla born?” Passage: “Nikola Tesla was born in Smiljan...” → “Smiljan”. |
| SQuAD v2.0 — Stanford Question Answering Dataset with unanswerable questions | Extractive question answering with no-answer detection | Question + passage | Answer span or no answer | Question asks for a date not stated in the passage → no answer. |
| SWAG — Situations With Adversarial Generations | Commonsense-style continuation selection | Context + four candidate endings | Best ending | Context: “A person opens the fridge.” Ending: “They take out food.” beats implausible alternatives. |
Some names are especially important for reading the paper. asks whether one sentence follows from another. asks whether two texts mean the same thing. assigns labels to tokens. asks BERT to find answer spans. Together, these tasks test very different uses of the same representation backbone.
What the experiments prove
The experiments are best read as a proof of the BERT recipe. The point is not simply that the tables contain high numbers. The point is that a single architecture pattern works across very different task formats with only small output changes.
| Experiment area | Main result | Interpretation |
|---|---|---|
| GLUE | BERTBASE and BERTLARGE outperform prior systems across the benchmark. | BERT is not a one-task trick; it transfers broadly across language-understanding tasks. |
| SQuAD v1.1 | BERT reaches very strong span-extraction results, with BERTLARGE setting a new top result. | Bidirectional token representations are especially useful for question answering. |
| SQuAD v2.0 | BERT handles unanswerable questions by using [CLS] as the no-answer span. | The same span-prediction machinery can be extended to answerability decisions. |
| SWAG | BERTLARGE strongly improves over ELMo-based and GPT baselines. | BERT's sentence-pair format also works for multiple-choice commonsense-style reasoning. |
is the broad test: BERT improves tasks involving inference, similarity, sentiment, paraphrase, and grammatical acceptability. is the token-level test: the model must identify exact spans, not just classify whole sentences. is the multiple-choice continuation test: the model must score candidate endings against a context.
The comparison with GPT is especially revealing. BERTBASE was intentionally sized close to OpenAI GPT, so the improvement cannot be explained only by saying "BERT is bigger." The paper argues that the key difference is the attention mask and objective: GPT is left-to-right; BERT is bidirectional through MLM.
Ablation studies
The ablations ask what part of the recipe actually matters. If BERT only worked because of more data or more parameters, removing NSP or replacing MLM with left-to-right language modeling would not be so informative. But the ablation results show clear patterns.
| Variant | What changes | Takeaway |
|---|---|---|
| BERTBASE | Full MLM + NSP + bidirectional encoder. | The reference point for the ablation. |
| No NSP | Remove the next sentence prediction objective. | Performance drops most clearly on tasks involving text-pair relations, especially QNLI and SQuAD. |
| LTR & No NSP | Use a left-to-right language model instead of MLM, with no NSP. | Large drops show that unidirectional pre-training is weaker for understanding, especially token-level QA. |
| + BiLSTM | Add a randomly initialized BiLSTM on top during fine-tuning. | Some right context is recovered for SQuAD, but it still cannot replace deep bidirectional pre-training. |
| Larger models | Increase layers, hidden size, and attention heads. | Bigger pretrained models improve downstream tasks, even when labeled data is small. |
| Feature-based BERT | Freeze BERT and use hidden states as features for NER. | BERT is strongest with fine-tuning, but its representations are also useful as fixed features. |
The most important ablation is the left-to-right variant. It performs much worse on SQuAD, which is exactly where the theory predicts trouble: token-level answer representations need right context. Adding a randomly initialized BiLSTM helps SQuAD but does not catch up, because it is not the same as pretraining every layer with bidirectional context from the start.
The model-size ablation also mattered historically. It showed that once a model is sufficiently pretrained, larger encoders can help even on small downstream datasets. This was one early sign that scale and pretraining could change the usual fear that large models simply overfit small NLP tasks.
Implementation details
BERT's main idea is simple, but the implementation recipe matters. The model is not just a bidirectional encoder in the abstract; it is trained with a specific data construction process, masking strategy, sequence length schedule, and fine-tuning setup. These details make the method reproducible and help explain why the original results were strong.
| Detail | Value / explanation |
|---|---|
| Pre-training data | BooksCorpus plus English Wikipedia, about 3.3B words total. |
| Sequence construction | Two text spans are packed together up to 512 tokens; 50% are real next-sentence pairs and 50% are random pairs. |
| Masking | 15% of WordPiece tokens are prediction targets; selected targets use the 80/10/10 replacement strategy. |
| Batch and steps | 256 sequences per batch, 1M training steps, about 128K tokens per batch at full length. |
| Optimization | Adam, learning rate 1e-4, warmup over 10K steps, linear decay, weight decay 0.01, dropout 0.1. |
| Activation | GELU instead of ReLU, following OpenAI GPT. |
| Sequence length trick | Train with length 128 for 90% of steps, then length 512 for the final 10% to reduce quadratic attention cost. |
| Fine-tuning search | Typical batch sizes 16 or 32, learning rates 5e-5 / 3e-5 / 2e-5, and 2–4 epochs. |
The most important implementation detail is that BERT does not predict every token. It selects a small subset of WordPiece tokens as prediction targets, corrupts some of them, and computes the MLM loss only at those selected positions. This makes MLM less prediction-dense than a left-to-right language model, but it gives the model something more valuable for understanding: every selected token can be recovered using both left and right context.
The sequence-length schedule also matters. Full self-attention becomes expensive as the sequence gets longer, so BERT is mostly pretrained with shorter sequences and only later trained with the full 512-token length. This keeps pre-training more efficient while still teaching the model how to use long-range position embeddings.
The architecture gives BERT the capacity to connect every token with every other token. The masking procedure prevents the model from seeing the answer directly. The large text corpus gives it broad language exposure. Fine-tuning then turns that general representation into a task-specific model. The recipe works because these pieces support each other.
Impact and afterlife
BERT's impact was both scientific and practical. Scientifically, it made masked encoder pre-training a standard baseline for language understanding. Practically, it gave NLP engineers a reusable pattern: take a pretrained checkpoint, convert the task into a BERT input format, add a small head, and fine-tune.
| Period | What changed |
|---|---|
| 2013–2017 | Word embeddings, seq2seq, attention, and ELMo show that learned representations and context matter. |
| 2017 | The Transformer replaces recurrence with attention as the main sequence-modeling engine. |
| 2018 | OpenAI GPT demonstrates Transformer pre-training plus fine-tuning; BERT demonstrates deep bidirectional encoder pre-training. |
| 2019–2020 | RoBERTa, ALBERT, DistilBERT, and many variants refine the recipe, challenge NSP, compress models, or scale training. |
| 2020s | Encoder models remain important for search, retrieval, classification, reranking, extraction, and embeddings, while decoder-only GPT-style models dominate open-ended generation. |
BERT also clarified a split that still matters. Encoder models are excellent when the goal is to understand, classify, rank, embed, retrieve, or extract from given text. Decoder-only models are better suited to open-ended generation. Modern systems often combine these families: a BERT-like encoder or embedding model may retrieve and rank evidence, while a GPT-like decoder writes the final response.
Why GPT became more visible
A natural question after reading BERT is: if BERT was so important, why do most people hear about GPT products instead of BERT products? The short answer is that generation is easier to package as a consumer product, while understanding is usually hidden inside infrastructure.
GPT-style models are trained to continue text from left to right. That objective maps directly onto products people can see: chatbots, writing assistants, coding assistants, copilots, and agents. A user types a prompt, and the model produces a visible answer. The model's core ability is already the interface.
BERT-style models have a different product shape. BERT reads a piece of text and turns it into a contextual representation. That representation is useful for judging relevance, assigning labels, extracting spans, matching a query to a document, or ranking candidates. These are extremely valuable commercial functions, but they usually happen behind the scenes.
| Model family | Core strength | Common commercial use | Product shape |
|---|---|---|---|
| BERT-like encoder | Reads and represents text | Search ranking, classification, extraction, reranking, embeddings | Invisible backend infrastructure |
| GPT-like decoder | Generates text, code, and actions step by step | Chatbots, writing tools, coding assistants, agents, copilots | Visible user-facing product |
This is why BERT can be everywhere without feeling like a product. Search engines, recommendation systems, customer-support routing, content moderation, document retrieval, entity extraction, and reranking systems all need strong language understanding. Users usually see the final ranked result, label, or extracted answer, not the encoder model that helped produce it.
BERT reads well. GPT writes well. That sentence is simplified, but it explains the commercial difference. Reading is often an internal capability inside a larger system; writing can become a standalone product interface.
BERT also shaped later systems even when the product name did not include "BERT." Many later encoder models and embedding models inherit the same broad idea: pretrain a language model to produce useful representations, then use those representations for search, ranking, classification, retrieval, and extraction. GPT became more visible because the user can talk to it directly, but BERT's legacy lives in the infrastructure that helps machines understand.
Takeaways
- BERT is an encoder-only Transformer designed for language understanding rather than open-ended generation.
- Its key training idea is MLM: hide or corrupt selected tokens, then predict the originals from bidirectional context.
- NSP adds a sentence-pair pre-training signal for the original BERT recipe.
- The input representation is the sum of token, segment, and position embeddings.
[CLS]provides a sequence-level representation;[SEP]separates text spans.- Fine-tuning means each downstream task gets its own task head and its own fine-tuned copy of the pretrained model.
- BERT's experiments show that one pretrained backbone can handle GLUE, SQuAD, SWAG, and token-level tasks with minimal architecture changes.
- The ablations show that deep bidirectional pre-training matters, especially for token-level question answering.
- BERT sits historically between ELMo and GPT: it keeps contextual representation and fine-tuning, but changes the directionality problem.
- Its afterlife includes many encoder variants and a lasting role in search, retrieval, classification, ranking, extraction, and embeddings.