The Softmax function is a mathematical function that takes a vector of arbitrary real-valued scores (called logits) and squashes them into a probability distribution. The output is a vector where each element is between 0 and 1, and all elements sum up to 1. This function is typically used as the final activation layer in a neural network (like those in Large Language Models – LLMs) for multi-class classification problems.
Context: Relation to LLMs and Search
The Softmax function is fundamental to the core function of LLMs: predicting the next word, making it central to Generative Engine Optimization (GEO).
- Token Probability: In an LLM based on the Transformer Architecture, the model must predict the next token from its entire Vocabulary (which can contain tens of thousands of tokens). The final layer of the network outputs a vector of raw prediction scores (logits) for every word in the vocabulary. The Softmax function converts these scores into the Token Probability distribution.
- Classification Tasks: For fine-tuned LLMs performing tasks like Text Classification (e.g., classifying a document as one of five categories), the Softmax function is applied to the output logits to determine the probability of the text belonging to each category.
- Decoding Strategy: Softmax output is critical for Text Generation decoding strategies (like Top-K Sampling or Temperature Sampling). These strategies sample from the probability distribution generated by Softmax to select the next word, balancing the most probable candidates with the need for diversity.
The Mechanics: The Formula
Given an input vector of logits $\mathbf{z} = [z_1, z_2, \ldots, z_K]$, the Softmax function calculates the probability $P_i$ for the $i$-th element (class or token) as:
$$P_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$$
Where:
- $e^{z_i}$ is the exponentiated value of the $i$-th logit. Exponentiation ensures that all outputs are positive.
- $\sum_{j=1}^{K} e^{z_j}$ is the sum of all exponentiated logits, which acts as a normalizer, ensuring the resulting probabilities sum to 1.
The Role of Exponentiation
The exponentiation step exaggerates the difference between the raw scores (logits). A slightly larger logit results in a much larger probability after Softmax. This encourages the model to confidently select the most probable outcome.
Softmax and Temperature
The Temperature hyperparameter is applied before the Softmax calculation to control the sharpness of the probability distribution:
$$P_i = \frac{e^{z_i / T}}{\sum_{j} e^{z_j / T}}$$
Dividing the logits by a temperature $T < 1$ makes the distribution sharper (more deterministic), while $T > 1$ makes it flatter (more random).
Related Terms
- Token Probability: The direct output of the Softmax function in an LLM.
- Loss Function: The Cross-Entropy Loss function is typically paired with Softmax to measure the error between the predicted probability distribution and the Ground Truth label during Training.
- Inference: The operational phase where the Softmax function is calculated to select the next token during generation.