Temperature is a Hyperparameter used in the decoding process of Large Language Models (LLMs) to control the randomness (or stochasticity) of the generated text. Formally, it is a scaling factor applied to the final layer’s logit outputs (raw prediction scores) before the Softmax function converts them into a probability distribution.
Context: Relation to LLMs and Search
Temperature is one of the most important parameters for tuning the behavior of an LLM, directly impacting the balance between creativity and factual adherence—a key consideration for Generative Engine Optimization (GEO).
- Controlling Predictability:
- High Temperature (e.g., $T > 1$): Increases the probability of low-likelihood tokens being selected. This results in more creative, diverse, and unexpected outputs, but also significantly increases the risk of Hallucination or incoherent text. This is often used for creative content generation.
- Low Temperature (e.g., $T < 1$): Decreases the probability of low-likelihood tokens and sharpens the distribution, concentrating probability mass on the few most likely tokens. This results in more deterministic, conservative, and factually grounded outputs. This setting is preferred for generating canonical Generative Snippets or coding tasks where accuracy is paramount.
- Zero Temperature ($T = 0$): This makes the output completely deterministic, equivalent to Greedy Search (always picking the single most probable token).
- GEO Strategy: A GEO specialist must choose a low Temperature setting for Retrieval-Augmented Generation (RAG) answers to ensure high Entity Authority and minimize ungrounded claims. Conversely, a higher temperature might be used for generating meta-descriptions or creative ad copy based on structured data.
The Mechanics: Softmax and Temperature
The Softmax function converts the raw logit scores ($z_i$) for each possible next token into a probability distribution ($P_i$). Temperature ($T$) is introduced into this function as a divisor for the logits:
$$P_i = \frac{e^{z_i / T}}{\sum_{j} e^{z_j / T}}$$
Where:
- $z_i$ is the raw logit score for token $i$.
- $T$ is the temperature value.
How Temperature Scales Probability
- Low Temperature ($T \rightarrow 0$): Dividing by a very small number exaggerates the differences between the logit scores, making the highest score’s probability approach 1, effectively ignoring all other tokens. The distribution becomes sharp.
- High Temperature ($T \rightarrow \infty$): Dividing by a large number shrinks the differences between the logit scores, causing all tokens (even low-probability ones) to have a more equal chance of being selected. The distribution becomes flat.
Temperature is often used in combination with other decoding methods like Top-K Sampling or Top-P Sampling (Nucleus Sampling) to further refine the selection process.
Related Terms
- Greedy Search: Equivalent to Temperature = 0.
- Inference: The operational phase where the Temperature setting is applied to control the output.
- Token Probability: The final likelihood value that Temperature is designed to manipulate.