Vocabulary

In a language model, the vocabulary is the full set of tokens the model knows. Each token has an embedding vector and a slot in the output layer.

Size is a tradeoff. A larger vocabulary means more parameters, because embedding matrices grow with the token count, and shorter sequences, because common words stay one token. Rare tokens show up less often, so they get a weaker learning signal. A smaller vocabulary means longer sequences, less memory, and more compute per word because words split into more pieces.

GPT-4 uses roughly 100,000 tokens. Earlier models used about 50,000. Multilingual models need larger vocabularies so they do not smash every language into tiny fragments. A specialized domain can justify a custom vocabulary trained on that corpus.

Special tokens do jobs, not words: [PAD] for padding, [BOS] and [EOS] for start and end, [UNK] for unknown, [SEP] to split segments, [MASK] for masked language modeling. After tokenizer training the vocabulary is frozen. Adding tokens later needs architecture work or a special path. If an important idea is missing from the vocabulary, the model has to build it from subwords.

That can cap how well it handles that domain. The vocabulary is the finite set of tokens the model knows. If a string is not in that set, the tokenizer splits it into pieces that are.