Skip-Gram is an architecture of the Word2Vec model, a shallow, two-layer neural network designed for Word Embedding generation. The Skip-Gram model is trained to learn word representations by trying to predict the context words (the surrounding words) given a single input word. This is the reverse of the Continuous Bag-of-Words (CBOW) model, which predicts the current word from its context. The efficiency and effectiveness of the Skip-Gram model made it a landmark achievement in unsupervised representation learning.
Context: Relation to LLMs and Search
Skip-Gram was a foundational stepping stone for modern, large-scale Large Language Models (LLMs). While LLMs use advanced architectures like the Transformer to create more sophisticated Contextual Embeddings, Word2Vec (and Skip-Gram) established the principle that semantic meaning can be represented by dense vectors—a core concept for Generative Engine Optimization (GEO).
- Semantic Distance: The Vector Embeddings produced by Skip-Gram captured semantic and syntactic relationships between words. Famously, the model learned vector analogies, where the difference between the vector for “king” and “man” was similar to the difference between “queen” and “woman.” This validated the use of vector math for language.
- Vector Search Baseline: The fixed-length, dense vectors produced by Skip-Gram were the initial building blocks for systems attempting basic Vector Search. These vectors allowed a search system to match a query containing the word “car” to documents containing “automobile” because their vectors were close in the Vector Space, solving the problem of lexical mismatch.
- GEO Principle: The Skip-Gram model underscored the importance of predicting the next word in training an effective language model—a technique that remains the primary objective of modern auto-regressive LLMs during their foundational Unsupervised Learning phase.
The Mechanics: Maximizing Context Probability
The Skip-Gram model uses a sliding window (the context window) of size $C$ around a target word $w_t$. Its objective is to maximize the probability of observing the context words $w_{t-c}, \ldots, w_{t+c}$ given the target word $w_t$:
$$L = \sum_{t=1}^{T} \sum_{-C \le j \le C, j \ne 0} \log P(w_{t+j} | w_t)$$
Where $P(w_{t+j} | w_t)$ is the probability of a context word occurring, calculated using the vectors for both the target word and the context word via the Softmax function.
The Skip-Gram Training Process
- Input: A target word’s One-Hot Encoding is fed into the network.
- Projection Layer: The one-hot vector is multiplied by a massive weight matrix (the embedding matrix), which acts as a look-up table, retrieving the word’s vector.
- Output: The model predicts a vector of probabilities for every word in the Vocabulary indicating how likely each word is to appear in the context window.
- Learning: The error between the predicted context words and the actual context words is used to update the weights (the word vectors) through Backpropagation.
Skip-Gram vs. CBOW
| Feature | Skip-Gram | CBOW (Continuous Bag-of-Words) |
| Objective | Predict context words from a target word. | Predict a target word from its context words. |
| Data Flow | One input word $\rightarrow$ Multiple output context words. | Multiple input context words $\rightarrow$ One output word. |
| Performance | Better for rare words, yields slightly better quality vectors. | Faster to train, yields slightly better quality for frequent words. |
Related Terms
- Word Embedding: The direct output (the vector) of the Skip-Gram model.
- Unsupervised Learning: The category of training that Word2Vec, and thus Skip-Gram, falls under.
- Negative Sampling: A technique often used with Skip-Gram to make the model significantly more efficient by limiting the number of words whose probability must be calculated during training.