Top-K

What top-k does to the model's choice

At every position, the model produces a probability for every token in its vocabulary. Sampling is the step that turns that distribution into one chosen token, and top-k is one of the rules that can narrow the field first.

The rule is simple enough to state in a sentence: keep the k most likely tokens, throw away everything else, rescale what remains so the probabilities add to one, then draw from that. Anthropic's reference describes the intent as sampling "only ... from the top K options for each subsequent token", used "to remove 'long tail' low probability responses." Google's puts it as considering "the set of topK most probable tokens."

Sampling parameters covers the family (temperature, top-p, and this one) and is the better starting point if you want the general picture. Top-k is covered in detail below, because it is in an unusual position right now.

A worked example with real probabilities

Numbers make the rule concrete. Suppose the model is completing "The capital of France is" and the top candidates come back like this:

With k = 3, the last two and the entire tail are discarded. The survivors (0.82, 0.06, 0.04) sum to 0.92, so each is divided by 0.92: Paris becomes 0.89, located 0.065, the 0.043. The ordering did not change. What changed is that everything below third place now has probability zero, and Paris picked up the mass they gave back.

Now take the same k at an open position, say after "Her favorite color was". The distribution there might be nearly flat across twenty plausible colors. k = 3 cuts seventeen reasonable answers.

That is the whole problem with the parameter, and it is visible in two examples.

Why a fixed count is a blunt instrument

Top-k applies a constant to something that is not constant.

The shape of the distribution changes at every position. Sometimes the model is certain and the useful set has one member; sometimes it is open and the useful set has fifty. A fixed k cannot be right for both. Set it low and you truncate the open positions, flattening variety where it was appropriate. Set it high and it does nothing at the confident positions, where it was not needed anyway.

Top-p was designed around this. Instead of a count it takes a share of probability mass and keeps the smallest set of tokens reaching it, so the set is small where the model is confident and large where it is not. It adapts to the shape; top-k ignores it.

That is why, when people compare the two, top-k tends to lose. It is also, for what it is worth, why top-k is the easier of the two to explain.

Where top-k stands at the three major providers

This is the part most writing on the subject gets wrong.

OpenAI does not offer it. The Chat Completions reference documents temperature and top_p. There is no top_k parameter. Code that sets one is sending a field the API does not read.

Anthropic has deprecated it. The Messages API reference carries a deprecation notice: "Models released after Claude Opus 4.6 do not accept top_k; any value will be rejected with a 400 error." Not ignored but rejected. top_p carries a matching notice, where "A value >= 0.99 will be accepted for backwards compatibility, all other values will be rejected with a 400 error." Both were already labeled "Recommended for advanced use cases only" before that.

Google makes it a per-model fact. The Model resource carries a topK field, and its documentation says: "If empty, indicates the model doesn't use top-k sampling, and topK isn't allowed as a generation parameter." So the answer varies by model, and the way to get it is to read the model's own record rather than the API reference.

All three checked on 2026-09-16, and all three are the kind of thing that moves, so check them again rather than trusting this paragraph in a year.

What to do when you meet top-k in existing code

You will meet it, because it appears in a great deal of sample code and in the settings of most local inference tools.

If you are calling a hosted API, the realistic answer is to leave it out. It is unavailable at one provider, rejected at another, and conditional at the third. Reach for temperature, which every provider supports and which sampling parameters explains, and change your prompt before you change your sampling.

If you find top_k set in a codebase, check when it was added. A value that has been there since a local prototype is probably inherited from a tutorial rather than chosen, and removing it changes less than the person who added it believed.

If you run models yourself, the calculation is different. Local runtimes still expose it, and self-hosted models is where this parameter still has a life. Even there, top-p adapts and top-k does not.

Further reading

• Anthropic, Messages API reference: the deprecation notices for top_k and top_p, in full.
• OpenAI, Chat Completions reference: the parameters that exist, and the ones that do not.
• Google, Models API reference: the topK field and what an empty value means.
• Sampling parameters: the family this belongs to, and why temperature is usually the one to reach for.