Tokenization

Tokenization is the process of converting raw text into a sequence of discrete tokens that serve as the fundamental units of input and output for language models. Tokens are not necessarily whole words; depending on the tokenization algorithm, they may represent words, subwords, individual characters, or byte sequences. The tokenizer is a critical component of any language model pipeline, as it determines how text is represented and directly impacts model performance, context window utilization, and inference costs.

Modern language models predominantly use subword tokenization algorithms. Byte Pair Encoding (BPE), used by GPT models, iteratively merges the most frequent character pairs to build a vocabulary of common subword units. WordPiece, used by BERT, takes a similar approach but optimizes for likelihood rather than frequency. SentencePiece provides a language-agnostic tokenization framework that operates directly on raw text without requiring pre-tokenization. The tiktoken library provides efficient tokenization for OpenAI models.

Tokenization has significant practical implications for AI applications. Context window limits are measured in tokens, not characters or words, so token-efficient representations allow more content to fit within the model context. Inference costs for API-based models are typically priced per token, making tokenization efficiency directly relevant to operating costs. Different languages tokenize with varying efficiency: English text typically requires fewer tokens per word than languages with complex morphology or non-Latin scripts, creating cost and performance disparities.

Understanding tokenization is essential for prompt engineering, cost estimation, and system design. Practitioners must account for tokenization when designing chunking strategies for RAG systems, managing context windows for long conversations, estimating API costs, and handling edge cases where tokenization boundaries affect model behavior. Specialized tokenizers may be needed for domains with unusual vocabularies, such as code, scientific notation, or multilingual content.