What is temperature in an LLM?

Explained

Temperature is a parameter applied when a language model generates text, controlling how much randomness goes into choosing each next token. The model assigns a probability to every possible next token; temperature reshapes that distribution before a token is sampled. A low temperature concentrates the choice on the most likely tokens and gives focused, repeatable answers; a higher temperature spreads it out and gives more varied, creative and less predictable text. Choosing it is part of matching a model to the task, from factual extraction to brainstorming.

How it works

At each step the model produces a score for every token in its vocabulary. These scores are divided by the temperature and turned into probabilities. Dividing by a small number exaggerates the differences, so the top tokens dominate; dividing by a larger number flattens them, so less likely tokens get more of a chance.

At or near zero, generation becomes close to always picking the most likely token. Even then, outputs are not guaranteed to be identical across calls, because of how models are served.

Which value for which task

• Low values suit extraction, classification, structured output, code and answers that must stay close to the sources.
• Middle values suit general assistance and writing where some variation is welcome.
• Higher values suit brainstorming, naming and creative drafts, where variety matters more than consistency.
• Whatever the setting, test it: the effect differs between models, and the provider's default is often a reasonable start.

Temperature, top-p and top-k

Temperature changes the shape of the whole probability distribution. Top-k keeps only the k most likely tokens. Top-p, or nucleus sampling, keeps the smallest set of tokens whose probabilities add up to p, cutting off the unreliable tail.

They can be combined, but changing several at once makes behaviour hard to reason about; providers usually recommend adjusting one. Some providers have also stopped accepting custom sampling values on their newest models and recommend steering behaviour through the prompt instead, so check the documentation of the model you use.

Frequently asked questions

Further reading

• Prompt Engineering for Enterprise Generative AI

Related terms

• Extraction Turning unstructured text, such as emails, contracts or invoices, into structured data: entities, fields, relations and events.
• Parameter A parameter is a numerical value a model learns during training; large language models have billions of them, and their count gives the model's size.
• Token The basic unit of text a language model reads and writes: a word, part of a word or a symbol. In English, one token is roughly four characters.
• Agent Agenda An AI agent's live plan: the ordered next steps, open subtasks, blocked items and checkpoints it works through towards a goal.
• Agent Planner The component or prompt pattern in an AI agent that breaks a goal into a sequence of steps before any tool is run.
• Agent Skill A packaged capability for an AI agent: the instructions, allowed tools, examples and input and output formats it needs to perform one coherent task.