Top-k Sampling

Top-k sampling limits the next token to the k most probable items so rare tokens are less likely to derail text.

After the model scores every token, top-k keeps the k highest logits, sets the rest to negative infinity, then runs softmax over those k tokens and samples from that truncated distribution. The integer k is the diversity versus quality tradeoff. k=1 is greedy decoding: always pick the top token. It is deterministic and often repetitive. k=50 allows moderate variety.

k=1000 allows more invention and also more incoherent jumps.

The flaw is that k is fixed even when the distribution shape changes. If one token has 95% probability, k=50 still admits 49 near-zero tokens that add noise. If many tokens share similar probability, k=50 may drop plausible options. Top-p, or nucleus sampling, moves the cutoff with cumulative probability instead.

Production systems often combine them: top-k as a hard cap, top-p as adaptive truncation inside that cap.

Implementation is sort logits, keep the top k ids, mask the rest to negative infinity, softmax, then sample. Combined with top-p, systems often apply top-k first as a hard cap so a long tail cannot leak through, then apply nucleus truncation inside that cap. Greedy decoding is the k=1 corner. Creative writing often uses larger k.

Code generation often uses small k because one wrong token breaks syntax. Top-k sampling was popularized for story generation in 2018: sample only from the k most likely next tokens.