Sampling: Temperature, Top-k, Top-p
Sampling in LLMs: temperature, top-k and top-p — how to pick a token from a distribution. It shapes the style of the text and is a frequent interview question.
Intuition
After every generation step the transformer returns a vector of logits — one number per vocabulary token. Softmax turns them into probabilities: “mat” — 30%, “windowsill” — 18%, “table” — 12%, and so on. Then we decide: always take the most probable token, or “roll the dice” over this distribution.
The simplest option is greedy decoding: always take the argmax. Sounds reasonable, but in practice the text quickly degenerates: the model loops and repeats the same phrases (“I think that I think that I think…”). The reason is that the most probable token at each step is not the same as the most sensible sequence overall, and repeating what was already said is often locally “safe” for the model.
How it works: temperature
Temperature is dividing the logits by a number T before softmax:
As T → 0 the gaps between logits inflate and almost all mass goes to one token — we get greedy generation. At T = 1 the distribution stays “as trained”. As T → ∞ the differences wash out and the distribution tends to uniform — the model starts producing a random stream of tokens. Important: temperature does not literally add “creativity” to the model — it only redistributes mass among options the model already considered possible.
Top-k, top-p and min-p: cutting the tail
Even at a sensible temperature the vocabulary still has thousands of tokens with a tiny but nonzero probability. Occasionally the dice land on one of them — and the sentence drifts into nonsense. So the tail of the distribution is cut:
- Top-k — keep the k most probable tokens (e.g. 50), zero the rest and renormalize. Simple, but inflexible: when the model is confident, k = 50 is too many; when it is not, too few.
- Top-p / nucleus sampling — keep the smallest set of tokens whose total probability is ≥ p (e.g. 0.9). The “nucleus” size is adaptive: for a confident model it has 1–2 tokens, for an uncertain one — dozens. That is why top-p has almost displaced top-k.
- Min-p — keep tokens whose probability is at least a fraction of the maximum (e.g. 10% of pmax). Another adaptive variant, popular in open-source inference.
Frequency / presence penalties hit a different problem — repetition: the logits of already used tokens are lowered, and looping becomes more “expensive” for the model.
Beam search and practice
Beam search keeps several best prefixes and searches for the sequence with maximum total probability. For machine translation and speech recognition it works, but for open-ended generation — poorly: the maximally probable text turns out predictable, clipped and full of repeats, because human speech is not actually “maximally probable”.
Practical guidelines: for code, fact extraction and structured answers, drop temperature toward 0–0.3 — accuracy matters, not diversity. For brainstorming and creative writing, raise it toward 0.8–1.2 and enable top-p ≈ 0.9–0.95.
- “What does temperature do mathematically?” — it divides the logits before softmax: T < 1 sharpens the distribution, T > 1 flattens it; at the extremes — argmax and uniform.
- “Why is top-p better than top-k?” — the cutoff size adapts to the model's confidence instead of being fixed in advance.
- “Why does greedy degenerate?” — a locally probable token ≠ a globally good text; the typical symptom is repetition loops, cured by sampling and penalties.
- “What settings would you use for SQL generation?” — low temperature (≈0–0.2), optionally a lower top-p; diversity is harmful here.
All interview questions in this section: LLMs: Training and Inference — interview questions