Stochastic Gradient Descent (SGD)

Stochastic Gradient Descent (SGD) is an iterative optimization algorithm used to train machine learning models, particularly deep neural networks like Large Language Models (LLMs). It is a variation of the classic Gradient Descent algorithm. Unlike standard Gradient Descent, which calculates the Gradient (slope of the error function) using the entire Training Set, SGD approximates the gradient by using only a single randomly selected training example at each step. This process makes the updates much faster and introduces randomness that helps the model escape local minima during optimization.

Context: Relation to LLMs and Search

SGD is a foundational algorithm that defines the efficiency and scalability of training all modern deep learning models, making it critical for the infrastructure supporting Generative Engine Optimization (GEO).

• Speed and Scale: Training an LLM on billions of parameters and petabytes of data is impossible with standard Gradient Descent (which would require loading the entire dataset into memory for every update). SGD’s approach of using one (or a small batch) of examples makes the computation highly parallelizable and feasible on massive distributed hardware (GPUs/TPUs). This efficiency enables the large-scale Training of the Transformer Architecture.
• Escaping Local Minima: The inherent stochasticity (randomness) in the weight updates means the optimization path is noisier than standard Gradient Descent. This noise is beneficial because it prevents the model from getting stuck in poor, suboptimal solutions (local minima) in the complex error landscape, promoting better Generalization.
• Batching for Efficiency (Mini-Batch SGD): In practice, pure SGD (using one example) is rarely used because it is too noisy for modern hardware. Most modern LLM training uses Mini-Batch SGD, which calculates the gradient using a small, randomly sampled subset of the data (a batch, e.g., 32 or 64 examples). This balances the smooth convergence of full Gradient Descent with the speed and computational benefits of SGD.

The SGD Process

In an iterative machine learning loop, SGD performs the following steps for each individual training example (or mini-batch):

1. Forward Pass: Feed the example ($\mathbf{X}$) through the model to get a prediction ($\hat{Y}$).
2. Loss Calculation: Calculate the error between the prediction ($\hat{Y}$) and the Ground Truth ($\mathbf{Y}$) using a Loss Function.
3. Gradient Estimation: Use the loss to calculate the Gradient for just that example (or batch) via Backpropagation.
4. Weight Update: Adjust the model’s Weights in the direction opposite the gradient, scaled by the Learning Rate ($\alpha$).

$$\mathbf{W}_{\text{new}} = \mathbf{W}_{\text{old}} – \alpha \cdot \nabla J(\mathbf{W}; \mathbf{x}_i, y_i)$$

Where $\nabla J(\mathbf{W}; \mathbf{x}_i, y_i)$ is the gradient of the loss function $J$ based on a single training example $(\mathbf{x}_i, y_i)$.

Moving Beyond Basic SGD

While SGD is foundational, its convergence can be slow and unstable. Modern LLM training almost universally uses Adaptive Optimization Algorithms which build upon SGD by dynamically adjusting the learning rate for each parameter. The most common of these is the Adam Optimizer.

Related Terms

• Gradient Descent: The general optimization technique from which SGD is derived.
• Adam Optimizer: A highly efficient, adaptive variation of the SGD algorithm.
• Backpropagation: The method used to efficiently calculate the gradient required by SGD.