Weights

Weights are the numerical parameters within a neural network that are multiplied by the inputs (or activations from a previous layer) and adjusted during the training process. In the context of a Large Language Model (LLM), weights store the learned relationships, patterns, and knowledge extracted from the massive training corpus, effectively defining the model’s intelligence and function.

Context: Relation to LLMs and Search

Weights are the essence of a model’s Entity Authority and predictive power, making them the ultimate, albeit indirect, target of Generative Engine Optimization (GEO).

• Knowledge Storage: In a Transformer Architecture, weights are housed in the dense layers and, critically, in the Self-Attention Mechanisms. They represent the strength of connection between different concepts (tokens/entities). A high weight between “Taptwice” and “GEO” means the model has learned a strong, persistent connection between the brand and the service.
• Influence and Training: Content structured with high Information Gain influences the training and Fine-Tuning of models. By consistently publishing unique, authoritative data, a brand can, over time, subtly bias the weights in pre-trained and continually learning models (like those powering Google SGE) towards their own canonical facts, reducing Hallucination risk.
• Parameter Scale: Modern LLMs are defined by the sheer number of their weights (parameters)—often billions or trillions. The scale of these weights is directly proportional to the model’s capacity for deep contextual understanding, Few-Shot Learning, and sophisticated Inference.

The Mechanics: The Learning Process

The adjustment of weights is the fundamental mechanism of machine learning, governed by calculus and optimization algorithms.

1. Forward Pass: Input data is passed through the network, multiplied by the current weights, and transformed by Activation Functions to produce an output prediction.
2. Loss Calculation: The output is compared to the Ground Truth to calculate the Loss Function, which quantifies the error.
3. Backpropagation: The error is propagated backward through the network, and the Gradient of the loss with respect to each weight is computed.
4. Weight Update: An optimizer (like Adam Optimizer) uses the gradient to update the weights in the direction that minimizes the loss, scaled by the Learning Rate.

Mathematical Representation

In a simplified neural connection, the output $z$ is a weighted sum of inputs $x$:

$$z = \sum_{i} (x_i \cdot w_i) + b$$

Where:

• $x_i$: The input signal (or previous activation).
• $w_i$: The weight assigned to that input connection.
• $b$: The bias term.

The new weight $w_{i}^{\text{new}}$ is calculated during training as:

$$w_{i}^{\text{new}} = w_{i}^{\text{old}} – \eta \frac{\partial \mathcal{L}}{\partial w_i}$$

Where $\eta$ is the learning rate, and $\partial \mathcal{L} / \partial w_i$ is the gradient of the loss ($\mathcal{L}$) with respect to the weight $w_i$.

Related Terms

• Gradient Descent: The optimization algorithm used to adjust the weights.
• Bias: An auxiliary trainable parameter that shifts the activation function’s output.
• Inference: The process of using the final, learned weights to generate an output or prediction.