Kullback-Leibler (KL) Divergence (also known as information gain or relative entropy) is a non-symmetric measure used in probability theory and information theory to quantify how much one probability distribution ($P$) differs from a reference probability distribution ($Q$). In essence, it measures the information lost when distribution $Q$ is used to approximate distribution $P$.
It is a key component in advanced machine learning, where it is used as a Loss Function or a regularization term to ensure that a learned distribution (the model’s output, $Q$) stays close to a target distribution ($P$).
Context: Relation to LLMs and Alignment
KL Divergence is a vital mathematical concept used in two critical stages of Large Language Model (LLM) development: Reinforcement Learning from Human Feedback (RLHF) and advanced Vector Embedding training.
1. RLHF (Alignment and Regularization)
KL Divergence is the central component of the PPO (Proximal Policy Optimization) algorithm used in RLHF to align LLMs with human preferences.
- The Conflict: After initial Pre-training, the LLM (the Policy Model) is highly knowledgeable but may produce unsafe or unhelpful responses. RLHF Fine-Tunes the Policy Model to maximize a Reward Model score (to be more helpful/harmless).
- The Constraint: Aggressively optimizing for the reward can cause the Policy Model to drift too far from the original, stable language patterns learned during pre-training, leading to a loss of coherence and fluency.
- The Solution (KL Term): A term based on the KL Divergence between the new Policy Model ($P$) and the original Pre-trained Model ($Q$) is added to the reward function. This KL penalty constrains the fine-tuning process, preventing the new model from deviating too much from the original model. The optimization thus becomes a trade-off: maximize the human reward while minimizing the KL Divergence from the original model.
2. Variational Autoencoders (VAEs) and Generative Models
KL Divergence is also integral to the Variational Autoencoder (VAE) framework (a generative model). In VAEs, the loss function includes a KL Divergence term that forces the learned latent distribution to stay close to a simple, standard prior distribution (e.g., a normal Gaussian). This ensures the Latent Space is well-behaved and can be used for smooth, continuous generation.
The KL Divergence Formula
For two discrete probability distributions, $P$ and $Q$, the KL Divergence ($D_{KL}$) is calculated as:
$$D_{KL}(P \parallel Q) = \sum_{i} P(i) \log \left(\frac{P(i)}{Q(i)}\right)$$
Where:
- $P(i)$ is the probability of event $i$ in the true/reference distribution.
- $Q(i)$ is the probability of event $i$ in the approximating/learned distribution.
Key Properties:
- Non-Negative: $D_{KL}(P \parallel Q) \ge 0$. It is zero only if $P$ and $Q$ are identical.
- Non-Symmetric: $D_{KL}(P \parallel Q) \ne D_{KL}(Q \parallel P)$. This is why it is called a “divergence” and not a true distance metric.
Related Terms
- Loss Function: The general class of functions to which KL Divergence belongs when used as a penalty or objective.
- Cross-Entropy Loss: Closely related to KL Divergence; Cross-Entropy equals the sum of Entropy (of $P$) and KL Divergence (of $P \parallel Q$).
- Regularization: The technique of adding a term (like the KL penalty) to the loss function to prevent Overfitting or model drift.