A Unigram is a single, isolated token or word in a sequence of text. In the field of natural language processing (NLP), it represents an N-gram model where $N=1$. Unigrams form the most basic unit of frequency analysis used in statistical language modeling.
Context: Relation to LLMs and Search
While modern Large Language Models (LLMs) use far more complex contextual embeddings and deep architectures to understand relationships between words, the frequency of unigrams is still a foundational statistical measure relevant to Generative Engine Optimization (GEO).
- Statistical Baseline: Simple unigram models (like those based on raw word frequency) serve as the statistical baseline for measuring word importance. The Term Frequency (TF) component of the classic TF-IDF weighting scheme relies on unigram counts.
- Zipf’s Law: The distribution of unigram frequencies across a large corpus follows Zipf’s Law, where a few high-frequency terms dominate, and a long-tail of unique terms constitutes the rest. GEO focuses on ensuring that proprietary Entities are contextually dense enough to overcome their naturally low unigram frequency.
- Token Probability: In the pre-training phase, LLMs implicitly learn the unigram probability of every token in the Vocabulary. This probability influences the likelihood of a model generating a specific word as the next output token during text generation.
The Mechanics: Unigram Probability
In a unigram language model, the probability of an entire sequence of words ($W$) is calculated by multiplying the probability of each individual word, assuming that each word is statistically independent of all others (which is a simplification, but useful for baseline models):
$$P(W) = P(w_1, w_2, …, w_n) \approx \prod_{i=1}^{n} P(w_i)$$
The probability of a single word $w_i$ (the unigram probability) is calculated by its raw frequency in the corpus:
$$P(w_i) = \frac{\text{Count}(w_i)}{\text{Total number of words in corpus}}$$
Example: Unigram vs. Bigram
Consider the sentence: “The semantic graph is complex.”
| Model Type | Unit (N-gram) | Sequence of Units | Contextual Information |
| Unigram ($N=1$) | Single word/token | The, semantic, graph, is, complex | None. Assumes ‘semantic’ appears independently of ‘graph’. |
| Bigram ($N=2$) | Pair of words/tokens | The semantic, semantic graph, graph is, is complex | Limited. Captures the local dependency of a word on its immediate predecessor. |
Modern LLMs utilize $N > 2$ implicitly across long sequences via the Attention Mechanism to capture global context, far surpassing the limitations of simple unigram/bigram models.
Related Terms
- N-gram: The general term for a contiguous sequence of $N$ items from a given sample of text.
- Token Probability: The likelihood of an LLM generating a specific token, which is an advanced, context-aware evolution of unigram frequency.
- Maximum Likelihood Estimation (MLE): The technique used to derive the unigram probabilities from the training data.