A Learning Curve is a plot or graph that tracks a machine learning model’s performance on a training dataset and a validation (or test) dataset as a function of the number of Training iterations (e.g., Epochs) or the size of the training data.
Learning curves are essential diagnostic tools for identifying and troubleshooting common model performance issues, most notably Overfitting, Underfitting, and determining whether adding more data will improve the model’s Generalization.
Context: Relation to LLMs and Model Health
Monitoring the learning curve is a mandatory practice for developing and Fine-Tuning Large Language Models (LLMs) because it provides the only reliable view into the model’s health and convergence.
- Diagnosis of LLM Training: The curves are typically plotted with the Loss Function (e.g., Cross-Entropy Loss) on the y-axis and the training steps or epochs on the x-axis. Observing the relationship between the training loss and the validation loss reveals the model’s primary problem:
| Curve Behavior | Problem Indicated | Suggested Solution |
| High Training Loss & High Validation Loss | Underfitting (Model is too simple) | Use a deeper, more complex Model Architecture (e.g., more Transformer layers). |
| Low Training Loss & High Validation Loss | Overfitting (Model memorized the training data) | Increase regularization (e.g., Dropout), use a larger Training Set, or stop training early. |
| Training Loss $\approx$ Validation Loss (Low) | Optimal Fit (Model generalizes well) | Training is complete and successful. |
- Detecting Data Sufficiency: A second type of learning curve plots performance against the size of the Training Set. If both the training and validation curves are leveling off and still show a large gap (high Overfitting), adding more data may not help. However, if both curves are still decreasing, it suggests that the model is data-hungry, and providing more data could yield significant performance improvements.
- Early Stopping: The learning curve is the mechanism that determines Early Stopping—the practice of halting training the moment the validation loss begins to increase, even if the training loss is still decreasing. This prevents the model from progressing further into the Overfitting regime, ensuring better Generalization.
Related Terms
- Overfitting: The most common problem diagnosed by learning curves, where the model performs excellently on the training data but poorly on unseen data.
- Underfitting: The problem of a model being too simple to learn the patterns in the data, visible as consistently high training and validation loss.
- Epoch: The unit of progress often used on the x-axis of a learning curve, representing one full pass through the entire Training Set.