A Multi-Layer Perceptron (MLP), also known as a Feed-Forward Neural Network (FFNN), is the foundational and most classical type of Neural Network in deep learning. An MLP consists of multiple layers of nodes (neurons) organized in a directed acyclic graph, where information flows strictly in one direction: from the input layer, through one or more hidden layers, and finally to the output layer.
MLPs are distinguished by their fully connected structure, meaning every node in one layer connects to every node in the next layer. Despite their simplicity, they serve as the building blocks for much more complex architectures, including modern Transformer Architecture models.
Context: Relation to LLMs and Deep Learning
Although the Transformer Architecture is the dominant framework for Large Language Models (LLMs), the MLP is not obsolete; it forms the critical Feed-Forward Network (FFN) component within every single Transformer Block.
- The Core Computation in Transformers: Within a Transformer‘s architecture, after the Attention Mechanism calculates the relationships between Tokens, an MLP is applied independently to the Vector Embedding of every token. This is typically a two-layer MLP where the first layer expands the token vector into a higher-dimensional space, and the second layer contracts it back to the original dimension. This step is crucial for transforming the token’s representation and allowing the model to learn complex, non-linear patterns within each token’s processed context.
- Universal Approximator: MLPs are mathematically proven to be Universal Function Approximators. This means that an MLP with just one hidden layer can theoretically learn to approximate any continuous function, provided it has enough neurons. This theoretical power is what enables the complex Semantics and patterns necessary for tasks like Natural Language Understanding (NLU) to be learned by an LLM.
- Historical Significance: Before the rise of Convolutional Neural Networks (CNNs) for vision and Recurrent Neural Networks (RNNs) for sequences, MLPs were the primary deep learning network used for tasks like image Classification and basic regression.
MLP Architecture and Function
An MLP is composed of three main types of layers:
- Input Layer: Receives the raw data (e.g., pixel values, or a Vector Embedding). It has no computational function other than distributing the input.
- Hidden Layers: One or more layers that perform the actual computation. Each neuron in the hidden layer calculates a weighted sum of its inputs and then passes the result through a non-linear activation function (e.g., ReLU or GELU, which is common in Transformers). This non-linearity is what allows the network to learn complex relationships.
- Output Layer: Produces the final result of the network (e.g., a probability distribution for a Classification task, or a single continuous value for a regression task).
Training an MLP is done using backpropagation, an algorithm that uses the Gradient Descent and chain rule to efficiently update all the network’s internal Weights based on the error calculated by the Loss Function.
Related Terms
- Neural Network: The general class of models that MLP belongs to.
- Transformer Architecture: The modern LLM architecture that uses MLPs as a core component.
- Activation Function: The non-linear function applied at each neuron, enabling the MLP to learn complex patterns.