One-sentence Summary

Google AI Language introduces BERT, a language representation model that pretrains deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, can be finetuned with one additional output layer without substantial task-specific architecture modifications, and obtains new state-of-the-art results on eleven natural language processing tasks, including GLUE ($80.5\%$80.5%), MultiNLI accuracy ($86.7\%$86.7%), SQuAD v1.1 Test F1 ($93.2$93.2), and SQuAD v2.0 Test F1 ($83.1$83.1).

Key Contributions

• The paper introduces BERT, a Transformer-based language representation model that pretrains deep bidirectional representations from unlabeled text by jointly conditioning on left and right context in all layers through a masked language model objective.
• The paper shows that pre-trained bidirectional representations can reduce the need for heavily engineered task-specific architectures, because a single BERT model can be fine-tuned with one additional output layer for sentence-level and token-level NLP tasks.
• The paper reports state-of-the-art results on eleven NLP tasks, including GLUE at 80.5%, MultiNLI at 86.7%, SQuAD v1.1 question answering Test F1 at 93.2, and SQuAD v2.0 Test F1 at 83.1, and notes that code and pre-trained models are available.

Introduction

Language model pre-training has become a key technique for improving natural language processing tasks that operate at both sentence level and token level, such as natural language inference, question answering, and named entity recognition. Prior methods mainly used feature-based or fine-tuning approaches with unidirectional language models, which limited the architectures that could be used during pre-training. For example, GPT trained models left to right, which can hurt performance on tasks requiring context from both directions, such as token-level prediction. The authors address this limitation with BERT, a deep bidirectional Transformer pretrained using a masked language model and a next sentence prediction task. This allows the model to combine left and right context during pre-training and supports a single fine-tuned representation that performs strongly across many NLP tasks.

Dataset

• The authors use two large English monolingual corpora for pre-training: BooksCorpus, with about 800 million words, and English Wikipedia, with about 2.5 billion words. From Wikipedia, they keep only text passages and remove lists, tables, and headers. They use document-level text rather than shuffled sentence collections so that training examples can contain long contiguous sequences.

• For pre-training, the authors construct masked language modeling examples from WordPiece tokens. They randomly select 15% of token positions in each sequence. The selected tokens are replaced with [MASK] 80% of the time, with a random token 10% of the time, and left unchanged 10% of the time, while the original token is used as the prediction target.

• The authors also construct next-sentence prediction labels. They pair sentence A with sentence B, where 50% of the time B is the true next sentence from the document and labeled IsNext, and 50% of the time B is a random sentence from the corpus and labeled NotNext. These inputs are packed into sequences with A and B sentence roles.

• For question answering, the authors use SQuAD v1.1, a dataset of about 100,000 crowdsourced question-answer pairs built from Wikipedia passages. Each example includes a question, an answer passage, and an answer text span. During fine-tuning, the question is packed as the A segment and the passage as the B segment; the dataset is used to supervise token positions that mark the start and end of the answer span.

• The authors also mention TriviaQA fine-tuning data as optional augmentation before SQuAD. They report that removing TriviaQA causes only a small performance loss on SQuAD, so the core dataset dependency remains SQuAD itself.

Method

The authors introduce BERT, a framework consisting of two main steps: pre-training and fine-tuning. During pre-training, the model is trained on unlabeled data using specific unsupervised tasks. For fine-tuning, the BERT model is initialized with the pre-trained parameters, and all parameters are subsequently fine-tuned using labeled data from downstream tasks. A distinctive feature of this approach is its unified architecture across different tasks, with minimal difference between the pre-trained and final downstream architectures.

Model Architecture The model architecture of BERT is a multi-layer bidirectional Transformer encoder. The authors denote the number of layers as $L$L, the hidden size as $H$H, and the number of self-attention heads as $A$A. They primarily report results on two model sizes: BERT$_{BASE}$BASE​ ($L=12, H=768, A=12$L=12,H=768,A=12, Total Parameters=110M) and BERT$_{LARGE}$LARGE​ ($L=24, H=1024, A=16$L=24,H=1024,A=16, Total Parameters=340M). Critically, the BERT Transformer uses bidirectional self-attention, unlike the constrained self-attention used in models like OpenAI GPT where every token can only attend to context to its left.

Input and Output Representations To handle a variety of downstream tasks, the input representation is designed to unambiguously represent both a single sentence and a pair of sentences in one token sequence. The authors use WordPiece embeddings with a 30,000 token vocabulary. The first token of every sequence is a special classification token ([CLS]), and the final hidden state corresponding to this token is used as the aggregate sequence representation for classification tasks. Sentence pairs are packed together into a single sequence, separated by a special token ([SEP]), and a learned embedding is added to every token indicating whether it belongs to sentence A or sentence B. For a given token, its input representation is constructed by summing the corresponding token, segment, and position embeddings.

Pre-training BERT The authors pre-train BERT using two unsupervised tasks rather than traditional left-to-right or right-to-left language models.

Task 1: Masked LM. To train a deep bidirectional representation, the authors mask some percentage of the input tokens at random and then predict those masked tokens. In all experiments, 15% of all WordPiece tokens in each sequence are masked at random. To mitigate the mismatch between pre-training and fine-tuning, since the [MASK] token does not appear during fine-tuning, the training data generator chooses 15% of the token positions at random for prediction. If the $i$i-th token is chosen, it is replaced with the [MASK] token 80% of the time, a random token 10% of the time, and the unchanged $i$i-th token 10% of the time. The final hidden vectors corresponding to the mask tokens are fed into an output softmax over the vocabulary.

Task 2: Next Sentence Prediction. To train a model that understands sentence relationships, the authors pre-train for a binarized next sentence prediction task. When choosing sentences A and B for each pre-training example, 50% of the time B is the actual next sentence that follows A (labeled as IsNext), and 50% of the time it is a random sentence from the corpus (labeled as NotNext).

Fine-tuning BERT Fine-tuning is straightforward since the self-attention mechanism in the Transformer allows BERT to model many downstream tasks by swapping out the appropriate inputs and outputs. For each task, the authors simply plug in the task-specific inputs and outputs into BERT and fine-tune all the parameters end-to-end. At the input, sentence A and sentence B from pre-training are analogous to sentence pairs in paraphrasing, hypothesis-premise pairs in entailment, question-passage pairs in question answering, and a degenerate text-$\emptyset$∅ pair in text classification or sequence tagging. At the output, the token representations are fed into an output layer for token-level tasks, such as sequence tagging or question answering, and the [CLS] representation is fed into an output layer for classification tasks. Compared to pre-training, fine-tuning is relatively inexpensive.

Experiment

The experiments fine-tune BERT on diverse NLP tasks, including GLUE, question answering, and commonsense inference, showing that simple task-specific adaptation yields strong performance across benchmark suites. Ablation studies indicate that deep bidirectional pretraining and larger model size are key to improving downstream accuracy, especially for lower-data tasks. Additional work on feature-based BERT representations demonstrates effectiveness for sequence labeling when full fine-tuning is not used, supporting BERT's versatility for both fine-tuning and feature extraction.

The reported GLUE results show that both BERT variants outperform prior systems, with the larger model reaching the best average performance among the compared methods. The gains are consistent across tasks and support the finding that larger pretrained models help even when downstream datasets are small. Both BERT variants beat OpenAI GPT and the prior state of the art systems on the averaged score. The larger BERT model is stronger than the smaller BERT model across every task in the comparison. Large pretrained models improve results on small GLUE datasets such as CoLA, MRPC, and RTE.

SQuAD v1.1 results compare answer span exact match and F1 scores for human performance, top leaderboard ensembles, and a published single model baseline. The BERT submission is described as a seven model ensemble that varies pretraining checkpoints and fine tuning seeds. The paper notes that some leaderboard entries may use unrestricted public data, so the system is also fine tuned on TriviaQA before SQuAD. The top ensemble entries exceed the listed human baseline on exact match and remain close on F1. The BERT approach uses a seven model ensemble rather than a single fine tuned checkpoint.

On SQuAD 2.0, which adds unanswerable questions, the approach represents a null answer with the [CLS] token and chooses a prediction threshold on development data to compare null and answer spans. It reports a clear gain over the best prior non-BERT system, while earlier leaderboard entries remain well below the human baseline. SQuAD 2.0 is made more realistic by allowing questions without a paragraph answer. No answer predictions are made by comparing the best answer span score with a null span score. The reported model improves over the previous best system by about five F1 points.

Fine-tuned BERT models achieve clearly higher SWAG accuracies than earlier embedding and ELMo based systems. BERT LARGE gives the best model performance and reaches human-level range, exceeding one expert setting while remaining below the multi annotator human result. The larger BERT model outperforms the smaller BERT variant and prior SWAG baselines. Model accuracy is competitive with human performance, though multiple human annotations remain stronger.

Comparing BERT_BASE variants shows that deep bidirectional masked training is stronger than a left-to-right training objective across GLUE and SQuAD tasks. The ablations also show that next sentence prediction helps question answering, natural language inference, and paraphrase-style reasoning, while a random BiLSTM over the left-to-right model only partially recovers passage understanding. Removing next sentence prediction reduces performance mainly on QNLI, MNLI, and SQuAD 1.1. A left-to-right no-NSP model is weaker than the masked bidirectional model on every reported task, with the largest gaps on MRPC and SQuAD. Adding a randomly initialized BiLSTM during fine-tuning raises SQuAD scores for the left-to-right model but leaves paraphrase and other GLUE results below the bidirectional BERT model.

The experiments evaluate BERT on GLUE, SQuAD v1.1, SQuAD 2.0, and SWAG, with GLUE showing consistent gains over prior systems and greater benefits from larger models, especially on small datasets. SQuAD results indicate that an ensemble fine-tuned model performs strongly on answerable questions, while a [CLS]-based null prediction mechanism improves unanswerable question handling; SWAG accuracy is competitive with human-level performance, though multi-annotator results remain stronger. Ablation studies further confirm that deep bidirectional masked training is important across tasks, and next sentence prediction mainly benefits question answering, natural language inference, and paraphrase-style reasoning.