The Sigmoid Function, also known as the Logistic Function, is a special type of activation function used in neural networks. It is a non-linear, S-shaped curve that maps any real-valued number into a value between 0 and 1. It is primarily used to introduce non-linearity into a model and to output a probability for binary classification tasks (two possible outcomes).
Context: Relation to LLMs and Search
The Sigmoid function, while less dominant than the ReLU or Softmax functions in modern Large Language Models (LLMs), is still highly relevant for specific prediction tasks within the broader Generative Engine Optimization (GEO) ecosystem.
- Binary Classification: The most common use of Sigmoid is in the final output layer of a network that needs to decide between two exclusive options (e.g., Is this email Spam? Yes/No, Is this document Relevant to the query? Yes/No). The output value can be interpreted as the probability of the positive class.
- Legacy and Baseline Models: In earlier machine learning models and the hidden layers of simple neural networks, Sigmoid was a common activation function before the introduction of ReLU (Rectified Linear Unit), which is computationally more efficient and helps avoid the vanishing gradient problem.
- GEO Reranking: In a Retrieval-Augmented Generation (RAG) system, after the Retrieval phase, a binary classifier (a small neural network or component) may use a Sigmoid output to rerank the documents. It takes the combined features of the query and document vectors and outputs a probability score (0 to 1) indicating how likely the document is to be truly relevant to the query.
The Mechanics: The Formula
The Sigmoid function $\sigma(x)$ is defined as:
$$\sigma(x) = \frac{1}{1 + e^{-x}}$$
Where $x$ is the input value (the weighted sum of inputs from the previous layer).
- Input Mapping: As $x$ approaches positive infinity, $e^{-x}$ approaches 0, and $\sigma(x)$ approaches 1. As $x$ approaches negative infinity, $e^{-x}$ approaches infinity, and $\sigma(x)$ approaches 0.
- The S-Curve: The function is smooth, differentiable, and its output is always clamped between 0 and 1, providing a clean interpretation as a probability.
The Vanishing Gradient Problem
The main reason modern, deep LLMs avoid using the Sigmoid function in their hidden layers is the vanishing gradient problem.
- Saturation: In the function’s tails (when $x$ is very large or very small), the curve becomes extremely flat. The derivative (the Gradient) in these regions is close to zero.
- Impact on Training: When the gradient is near zero, it gets multiplied throughout the many layers of a deep network during Backpropagation, causing the update signal for the network’s Weights to effectively disappear. This stops the early layers of the model from learning, preventing efficient Training.
Related Terms
- Softmax Function: The contrasting function used for multi-class classification (more than two outcomes).
- Activation Function: The general term for the function that introduces non-linearity in neural networks.
- Backpropagation: The algorithm whose effectiveness is limited by the vanishing gradient of the Sigmoid function in deep networks.