Tokenization

Tokenization is the fundamental process in natural language processing (NLP) of converting raw text into smaller, discrete units called tokens. These tokens are the numerical inputs that Large Language Models (LLMs) and other machine learning models actually process. Tokenization is the first step in any NLP pipeline and dictates how the model perceives language structure, meaning, and the overall size of its Vocabulary.

Context: Relation to LLMs and Search

Tokenization has a profound impact on the efficiency, memory consumption, and contextual understanding of LLMs, directly influencing Generative Engine Optimization (GEO).

• Numerical Representation: LLMs cannot process text directly; they operate on vectors of numbers. Tokenization maps words, parts of words, or characters to unique numerical IDs, which are then converted into Word Embeddings or Vector Embeddings.
• Context Window and Latency: The length of a user prompt or a retrieved document in an Retrieval-Augmented Generation (RAG) system is measured by the number of tokens. Efficient tokenization minimizes the total token count, allowing more semantic content to fit within the fixed Context Window and reducing Inference latency.
• GEO Strategy: For content to be effectively understood by AI Answer Engines, it must be structured in a way that minimizes token “waste” and ensures canonical Entities are represented by stable, high-quality tokens.

Types of Tokenization

There is a trade-off between Word-based Tokenization (large vocabulary, handles out-of-vocabulary words poorly) and Character-based Tokenization (small vocabulary, slow to process). Modern LLMs primarily use Subword Tokenization to achieve the best of both worlds.

1. Word Tokenization

Splits text based on spaces and punctuation.

• Pro: Simple, maintains natural word boundaries.
• Con: Large vocabulary, struggles with unseen words (Out-of-Vocabulary – OOV).
• Example: “Tokenization is fast” $\rightarrow$ [‘Tokenization’, ‘is’, ‘fast’]

2. Character Tokenization

Splits text into individual characters.

• Pro: Minimal vocabulary size, no OOV problem.
• Con: Sequences are very long, losing semantic meaning.
• Example: “fast” $\rightarrow$ [‘f’, ‘a’, ‘s’, ‘t’]

3. Subword Tokenization (The LLM Standard)

This method splits rare words into meaningful sub-word units, while keeping common words intact. The primary algorithms are Byte-Pair Encoding (BPE), WordPiece, and Unigram Language Model.

• Pro: Manages the vocabulary size effectively, handles compound and complex words (e.g., ‘geospatial’ $\rightarrow$ ‘geo’ + ‘spatial’), and minimizes the OOV problem.
• Con: Tokens are not always intuitively recognizable as full words, and subtle differences in text formatting can lead to different token representations.
• Example: “untokenizable” $\rightarrow$ [‘un’, ‘token’, ‘iz’, ‘able’]

The Tokenization Pipeline

1. Normalization: Cleaning text (e.g., lowercasing, removing HTML tags).
2. Segmentation: Splitting the text into potential tokens (e.g., based on white space).
3. Encoding: Applying the subword algorithm (like BPE) to convert the segments into final tokens.
4. Mapping: Converting the final tokens into their unique numerical token IDs for model processing.

Related Terms

• Token: The resulting unit of text after tokenization.
• Vocabulary: The set of all unique tokens known to the model.
• Context Window: The maximum number of tokens an LLM can process in a single sequence.