Leaky ReLU (LReLU) is a variation of the Rectified Linear Unit (ReLU), a non-linear Activation Function used in deep Neural Networks. LReLU is designed to solve the “dying ReLU” problem, where neurons become permanently inactive during Training.
While standard ReLU outputs zero for any negative input, LReLU introduces a small, positive slope (a “leak”) for negative inputs. This ensures that the neurons remain active, allowing the flow of Gradient Descent information during backpropagation.
Context: Relation to LLMs and The Dying ReLU Problem
LReLU’s design principle is crucial for the efficient Optimization of deep neural networks, although it is not the most common activation function in modern Large Language Models (LLMs).
- The Dying ReLU Problem: In standard ReLU, if the input to a neuron is negative, the output is zero. If the gradient flowing back through that neuron is also zero, the neuron’s Weights will never be updated, and the neuron effectively “dies” (becomes permanently inactive). This reduces the model’s capacity to learn.
- How LReLU Solves It: LReLU provides a non-zero, albeit very small, gradient for negative inputs. Even if a neuron constantly receives negative input, its Weights will still receive a small update during Training, keeping the neuron alive and allowing it to eventually move out of the negative range.
- LLM Activation Functions: While LReLU is an important improvement over the original ReLU, modern Transformer Architecture models (like GPT and BERT) commonly use the GELU (Gaussian Error Linear Unit) function. GELU is a smoother, probabilistic variant of ReLU that has proven to work even better than LReLU in high-dimensional deep language models by providing continuous and non-monotonic gradients. However, LReLU remains a powerful option, particularly in specialized deep learning applications like Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs).
The Leaky ReLU Formula
The Leaky ReLU function is defined as:
$$f(x) = \begin{cases} x & \text{if } x \ge 0 \\ \alpha x & \text{if } x < 0 \end{cases}$$
Where:
- $x$ is the input to the neuron.
- $\alpha$ (alpha) is the leakage rate (a small positive constant, typically set to $0.01$ or $0.1$).
A variant, the Parametric ReLU (PReLU), turns $\alpha$ into a learned Parameter that the network optimizes during Training, allowing the model to determine the optimal slope for the negative region itself.
Related Terms
- Activation Function: The general class of functions to which LReLU belongs, which introduces non-linearity.
- ReLU (Rectified Linear Unit): The function that LReLU was created to improve upon.
- Gradient Descent: The Optimization process that relies on the non-zero gradients provided by LReLU.