Sampling Parameters

Where sampling parameters sit in a model call

A language model does not pick a word. It produces a probability for every token it could produce next, across its whole vocabulary. Sampling is the step that turns that distribution into one actual token, and sampling parameters are the settings that govern the draw.

This matters because of what they cannot do. Temperature will not teach a model your refund policy, and lowering it will not make a wrong answer right. The distribution is fixed by the time these settings apply. You are choosing how to read it, not what it says.

How temperature reshapes the distribution

Temperature scales the distribution before the draw. Below 1 it sharpens: likely tokens get likelier, unlikely ones fall away, and the model keeps returning to the same phrasings. Above 1 it flattens, and tokens the model rated as long shots start appearing. OpenAI's reference gives the range as 0 to 2.

At 0 the model stops sampling and takes the highest-probability token every time. That is called greedy decoding, and it is the setting you want for extraction, classification, and anything your code parses.

How top-p and top-k truncate the distribution

Both cut the distribution down before the draw, using different rules.

top-p, also called nucleus sampling, keeps the smallest set of tokens whose probabilities add up to p. OpenAI describes 0.1 as considering "only the tokens comprising the top 10% probability mass." The size of that set changes per token: when the model is confident, a handful of candidates reach the threshold; when it is unsure, the set grows.

top-k keeps a fixed number of candidates, whatever their probabilities. It is simpler and less adaptive, and it is the one most likely to be missing or named differently, depending on your provider.

Do not reach for all three at once. OpenAI's own guidance is blunt: "We generally recommend altering this or temperature but not both." Stacking them makes the behavior hard to reason about and harder to explain to whoever debugs it next.

Setting temperature and top-p by task

For anything your code parses, pull temperature to 0 and leave the truncation settings alone. You want the same input to give you the same field.

For drafting, summarizing, or naming things, raise it. Somewhere near the middle of the range gives you variation without wandering. Test on your own prompts rather than adopting a number from a blog post, because the useful setting depends on how constrained your prompt already is.

When output degrades into repetition, the fix is usually the prompt rather than the knobs. A model that keeps restating the same sentence is often working from a prompt that gave it nothing else to say.

Why temperature 0 is not a reproducibility guarantee

Greedy decoding removes the randomness in the draw. It does not promise you the same bytes tomorrow, and OpenAI's reference says so outright:

Determinism is not guaranteed.

There is a seed parameter, and its wording is worth reading closely. The system "will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result." Best effort and should, not will.

Responses also carry a system_fingerprint, which "represents the backend configuration that the model runs with." Read it alongside seed to spot when the backend changed underneath you. That is the honest picture: temperature 0 narrows the draw, seed is a best effort on top of it, and the infrastructure underneath can still move.

Design around it. Assert on structure and meaning rather than exact strings, and treat any test that compares a full response to a stored copy as a test that will fail for reasons unrelated to your code.

Further reading

• OpenAI, Chat API reference: parameter ranges, the seed and fingerprint behavior, and the advice against changing temperature and top-p together.
• Inference: what else happens between your request and the answer.
• Large language models: where the distribution comes from, if the first section went past too quickly.