Top-p Sampling

Top-p sampling, or nucleus sampling, keeps the smallest token set whose probabilities add up to at least p.

After logits become probabilities, tokens are sorted high to low and included until the running sum reaches p. 9, only that token remains. If mass is spread across many tokens, more of them enter the set. That is the gain over top-k: the set expands when the model is unsure and shrinks when it is sure. 9 is a common default across tasks.

It keeps most of the useful probability mass and still allows occasional surprises. 7, yields tighter, more predictable text. 95 and above, yields more variation.

The 2019 paper "The Curious Case of Neural Text Degeneration" introduced top-p and showed it produced more human-like text than top-k or temperature alone. Most production stacks pair top-p with temperature: temperature changes the shape of the distribution, then top-p keeps the nucleus of likely tokens. That pairing is stable across many prompts.

Nucleus size is data-dependent. A peaked next-token distribution yields a tiny nucleus. A flat distribution yields a large one. 9 can include one token on some steps and dozens on others. Temperature still matters: it changes the probabilities that top-p then truncates.

The 2019 degeneration paper argued that a fixed k cannot track this shape change, which is why nucleus sampling spread in production. Nucleus (top-p) sampling was proposed in 2020. You sample from the smallest set of tokens whose probabilities sum to p.