Initialization

Initialization refers to the process of setting the initial values for the trainable Parameters (the Weights and Bias vectors) within a Neural Network (NN) before the start of the Training process.

Proper initialization is critical for deep learning models, including Large Language Models (LLMs), because it dictates the starting point for the Optimization algorithm (Gradient Descent). Poor initialization can lead to training failures like vanishing or exploding gradients, which prevent the model from learning effectively.

Context: Relation to LLMs and Training Stability

In the complex, deep architecture of Transformer Architecture models, initialization is arguably as important as the choice of Optimization algorithm and Learning Rate.

The Problem with Poor Initialization

Deep networks, such as those used for LLMs, have hundreds of Hidden Layers. During backpropagation, gradients are multiplied layer by layer:

• Zero Initialization: If all weights are initialized to zero, every neuron in every layer will produce the same output and receive the same gradient. This symmetry causes the network to learn only one feature, regardless of its depth (symmetry breaking is key).
• Too Large Initialization: If weights are too large, the output of each layer will grow exponentially as data moves forward (exploding activation values). When gradients are calculated during backpropagation, they will also explode (Exploding Gradients), making the optimization unstable.
• Too Small Initialization: If weights are too small, the output signals and the corresponding gradients will shrink exponentially as they pass through the deep network layers (Vanishing Gradients). The layers near the start of the network will receive almost no update, and the model will fail to learn.

Modern Initialization Techniques (LLMs)

Modern Large Language Models (LLMs) use sophisticated statistical methods to ensure that activation and gradient variances remain stable across all layers:

1. Xavier/Glorot Initialization: Designed for networks using activation functions like Sigmoid or tanh. It scales the initial weights based on the number of input and output neurons in a layer to maintain variance.
2. Kaiming/He Initialization: Specifically designed for networks using ReLU and its variants like Leaky ReLU (LReLU) (which are common in Transformer Architecture). It uses a different scaling factor to account for the zeroing out of negative inputs by ReLU, ensuring a consistent forward signal.
3. Specialized Transformer Initialization: LLMs often use minor variations of Kaiming initialization applied specifically to the linear layers in the Attention Mechanism and MLP (Multi-Layer Perceptron) to maintain training stability at massive scales.

Related Terms

• Weights: The trainable Parameters that initialization targets.
• Gradient Descent: The Optimization algorithm that relies on stable gradients provided by proper initialization.
• Vanishing Gradients: The primary failure mode that proper initialization is designed to prevent.