Tokenizer

A tokenizer converts raw text into a sequence of discrete tokens a language model can process, bridging human-readable text and numeric model inputs.

Different tokenizers split the same text differently. That changes behavior, speed, and what the model can do. The tokenizer defines the vocabulary and the split rules. Common methods are byte-pair encoding, WordPiece, and SentencePiece. All learn subword units that trade vocabulary size against sequence length.

75 words per token on average, but language, domain, and tokenizer all move that number. Languages with complex morphology or non-Latin scripts often use more tokens per word and burn more of the context window. Code, math notation, and technical writing can tokenize badly if they were scarce in tokenizer training.

If a distinction is not in the token split, the model cannot learn it. More tokens means more compute. Wasteful tokenization wastes the context window. Models struggle on text that tokenizes poorly. Training and inference must use the same tokenizer so the mapping stays consistent.

A word that becomes three tokens costs three attention steps instead of one. That is why inefficient tokenization wastes context and money. Subword methods such as BPE, WordPiece, and SentencePiece exist to keep common words short while still covering rare strings. If training used tokenizer A and inference uses tokenizer B, ids no longer match embeddings, and the model produces garbage.

Stick to one tokenizer for both. Byte Pair Encoding for translation was published in 2016. GPT-2, GPT-3, and many later models tokenize with a BPE variant.