Temperature, Top-P, and Top-K

Now we'll explore an important question:

How does an LLM decide which token to generate next?

Imagine you ask an AI:

Write one sentence about a beautiful morning.

It might generate:

• "The sun rose over the peaceful hills."

• "Golden sunlight filled the quiet garden."

• "A gentle breeze welcomed the new day."

Why might the answers differ? One reason is that the model can assign different probabilities to possible next tokens, and its generation settings influence how those options are selected.

Today, we'll learn three important settings:

Here, blue is the most likely next token, but other options also have some probability.

The model can select the most likely token using a greedy strategy, or sample from a probability distribution. Temperature, Top-P, and Top-K can influence sampling when those options are supported.

Let's understand each one separately.

2. Temperature — how predictable is the output?

Temperature is a setting that influences how a model distributes probabilities among candidate tokens when sampling.

Think of it like choosing between predictable and varied answers.

• Low temperature: the model generally favors the highest-probability options more strongly.

• High temperature: probability is distributed more evenly, making lower-ranked options more likely to be sampled.

Temperature does not directly change the model's knowledge. It changes the sampling behavior.

Imagine the prompt is:

Suggest a name for a friendly coding assistant.

0 to 0.3 - Low temperature

Likely behavior: More predictable choices.

Example name: CodeMate

The same prompt may produce more consistent results, although determinism is not guaranteed.

0.3 to 1.2 - Moderate temperature

Likely behavior: A balance of common and alternative choices.

Example name: ByteBuddy

Useful for everyday writing, brainstorming, and many general tasks.

1.2 to 2.0 - High temperature

Likely behavior: More varied or surprising choices.

Example name: PixelPhoenix

Can encourage variety, but may also produce less coherent or less useful results.

This is a conceptual demonstration, not a live model call. These names illustrate possible styles rather than guaranteed outputs. 

Important: Low temperature does not guarantee correctness. If an LLM doesn't know an answer, setting temperature to zero does not magically make it know the answer.

Also, a temperature of zero is often treated as greedy or near-deterministic generation, depending on the provider. It does not guarantee identical outputs across every model, backend, or request.

3. Top-P — nucleus sampling

Top-P is also called nucleus sampling. Rather than selecting candidates based on a fixed number, it keeps the smallest set of high-probability candidate tokens whose cumulative probability reaches a chosen threshold \(p\).

Let's return to our example: