Byte-Pair Encoding

Byte-pair encoding builds a subword vocabulary by merging the most frequent adjacent pairs again and again.

It starts from individual characters or bytes. It counts adjacent pairs in the training corpus and merges the most frequent pair into one new token. That loop repeats until the vocabulary hits the target size, typically 30,000 to 100,000 tokens. Common words collapse into single tokens through repeated merges. Rare words stay split into frequent subword pieces.

Any unseen word can still be written as a sequence of subwords, so the model can handle misspellings, new coinages, and technical terms it has never seen as whole tokens. The model also learns how those pieces relate.

BPE training depends on the corpus. A tokenizer trained on English does poorly on code or other languages. Once trained, merge rules are deterministic, so the same text always tokenizes the same way. GPT models use BPE variants. BERT uses WordPiece, which is similar but scores merges differently. Common words from the training corpus tokenize tightly.

Rare domains and languages tokenize loosely and use more tokens.

Merge order is stored as a list of pair rules. At inference the tokenizer applies those rules greedily to new text. A word seen often in the BPE corpus becomes one token. A chemical formula or a snippet of Rust may stay a long chain of byte-level pieces. That is the compression tradeoff the 30,000 to 100,000 vocabulary size is trying to set.

Rico Sennrich, Barry Haddow, and Alexandra Birch introduced BPE subwords for neural translation in 2016.