Token is the smallest unit that a language model reads and writes. Models do not directly recognize "characters" or "words"; instead, they first segment text into tokens, map them to integer IDs via a lookup table, and feed them into neural networks. The segmentation method directly determines the vocabulary size, sequence length, and computational cost: segmenting by whole words causes the vocabulary to expand with language variety and leads to "out-of-vocabulary" issues for new words; segmenting by characters results in excessively long sequences and fragmented semantic units.

The term "token" originally referred to the unit of lexical analysis in compiler theory, predating large models. In the era of statistical machine translation, this issue was mitigated by fallback rules such as dictionaries. In 2015, Sennrich, Haddow, and Birch systematically introduced subword segmentation based on Byte Pair Encoding (BPE) in their paper Neural Machine Translation of Rare Words with Subword Units. This approach retains common words as whole tokens while splitting rare words into smaller fragments. Starting from characters, it counts the frequency of adjacent symbol pairs and iteratively merges the most frequent pair until the vocabulary reaches a preset size; during translation, subwords are recombined. On WMT15 English-German and English-Russian tasks, this method improved BLEU scores by 1.1 and 1.3 points, respectively, over the baseline using fallback dictionaries, becoming a landmark work in applying subword tokens to neural networks.

Since then, BPE and other subword-based approaches like WordPiece and SentencePiece have become mainstream for large model tokenizers. The GPT series also employs BPE-style tokenizers, with vocabulary sizes typically ranging from tens of thousands to over a hundred thousand. Vocabulary size involves a trade-off: a larger vocabulary allows each token to carry more information and shortens sequences, but increases the size of the embedding matrix and output layer; a smaller vocabulary results in longer sequences and higher computational costs. Since the number of tokens directly affects API billing, context window usage, and inference overhead, "how many tokens a given sentence is segmented into" has become a critical engineering consideration requiring careful optimization.