A Loss Function (also known as a cost function or objective function) is a mathematical function that quantifies the difference (or “loss”) between the prediction made by a machine learning model and the actual true value of the data being predicted. The loss function is defined for a single training example or a Mini-Batch of data.
During the Training process, the primary goal of the model’s Optimization algorithm is to find the set of internal Weights that minimizes the value of this loss function. This minimization process, often driven by Gradient Descent, forces the model to learn patterns that lead to highly accurate predictions.
Context: Relation to LLMs and Training
The Loss Function is the compass that guides the entire Optimization of Large Language Models (LLMs), including their massive Pre-training phase and subsequent Fine-Tuning.
- The Core LLM Loss (Cross-Entropy): For the fundamental LLM task of next-token prediction (predicting the next word or Token), the standard loss function is Cross-Entropy Loss. This function measures the difference between the probability distribution predicted by the model for the next token and the true probability distribution (which is 1 for the correct next token and 0 for all others). Minimizing Cross-Entropy Loss is mathematically equivalent to maximizing the Maximum Likelihood of the training data.
- Loss for Regression (MSE): For tasks where the LLM or an associated search model must predict a continuous number (e.g., a Relevance score, a ranking value, or the distance between two Vector Embeddings), the Mean Squared Error (MSE) is commonly used.
- Loss for Retrieval (Margin Loss): In Metric Learning for Neural Search, specialized loss functions like Triplet Loss (which incorporates a Margin) are used to ensure that relevant documents are mapped significantly closer to the query in the vector space than irrelevant documents.
Loss Function vs. Objective Function
The terms Loss Function and Objective Function are often used interchangeably, but in formal terms:
- Loss Function ($L$): Measures the error for a single data point or a single batch.
- Cost Function ($J$): Measures the total average loss across the entire Training Set.
- Objective Function: The general term for the function being maximized or minimized by the algorithm. For training, the objective is to minimize the Cost Function.
Common Loss Functions
| Loss Function | Use Case | Mathematical Goal |
|---|---|---|
| Cross-Entropy Loss | Classification (e.g., next-token prediction, spam detection). | Measures divergence between predicted and true probability distributions. |
| Mean Squared Error (MSE) | Regression (e.g., predicting house prices, document scores). | Minimizes the average squared error between prediction and true value. |
| Triplet Loss | Metric Learning (e.g., creating Vector Embeddings). | Forces a Margin of separation between similar and dissimilar vectors. |
Related Terms
- Gradient Descent: The algorithm that uses the gradient (slope) of the loss function to adjust the model’s Weights.
- Optimization: The overall process guided by the minimization of the loss function.
- Objective Function: The broader term encompassing the loss function.