Term Frequency (TF) is a statistical measure used in natural language processing (NLP) and information retrieval to evaluate how often a specific word (or token) appears in a document. It quantifies the importance of a term within the context of that specific document.
Context: Relation to LLMs and Search
Term Frequency is a foundational concept that, while superseded by modern Vector Embeddings in advanced Large Language Models (LLMs), remains crucial for baseline search algorithms and the operational efficiency of Generative Engine Optimization (GEO).
- Relevance Baseline: TF is the first component of the classic TF-IDF (Term Frequency-Inverse Document Frequency) weighting scheme. Before Vector Search dominated, search engines relied heavily on TF to assess a document’s immediate relevance to a query. A high TF score for a query term suggested high relevance.
- Content Density: For GEO, TF is still a key metric for Content Engineering. Maximizing the TF of canonical Entities (e.g., a brand name, a proprietary product) is essential to ensure that the document’s subject matter is unambiguously clear to the indexing system and the LLM’s Attention Mechanism.
- Contrast with Semantic Models: While TF measures raw count, modern LLMs use Contextual Embeddings which measure the meaning of a term based on its surrounding words. For example, the word “apple” has different semantic importance in a document about fruit versus one about technology, regardless of its raw frequency.
The Mechanics: Calculation
The simplest way to calculate Term Frequency is the raw count, but this is often normalized to prevent bias towards long documents.
1. Raw Count
The count of term $t$ in document $d$:
$$TF(t, d) = \text{Count of term } t \text{ in document } d$$
2. Normalized (Augmented) Frequency
The most common normalization method divides the raw count by the maximum term frequency in the document. This is used to suppress the effect of extreme document length variations.
$$TF_{\text{normalized}}(t, d) = 0.5 + 0.5 \cdot \frac{\text{Count}(t, d)}{\max_{w \in d} \text{Count}(w, d)}$$
The factor of $0.5$ is added to ensure that a term that does not appear still has a non-zero TF (though the term is often 0 for absent words).
Example
Consider a document (D1) with 100 total words.
- Term $t_1$ (“Generative”): Appears 10 times.
- Term $t_2$ (“the”): Appears 20 times (the maximum frequency term).
- Raw TF ($t_1$): 10
- Normalized TF ($t_1$): $0.5 + 0.5 \cdot \frac{10}{20} = 0.5 + 0.25 = 0.75$
The TF-IDF Relationship
Term Frequency is often paired with Inverse Document Frequency (IDF) to create the TF-IDF score.
- High TF: Indicates the term is important to this document.
- High IDF: Indicates the term is rare and important to the entire corpus.
TF-IDF rewards terms that are frequent in a specific document but rare across the corpus, effectively filtering out common words (stop words) like “the” and “a” that have high TF but low value.
Related Terms
- Inverse Document Frequency (IDF): The statistical counterweight to TF, measuring a term’s rarity across the document collection.
- Unigram: The single word or token unit whose frequency is counted by the TF metric.
- Evaluation Metric: TF-IDF is an example of a weighting system used to determine the relevance score, which is a key evaluation metric in search.