Tokens

Tokens are the discrete units into which text is divided for processing by language models. They represent the fundamental input and output elements of modern AI systems: every piece of text a language model reads or generates is first converted into a sequence of tokens, and all internal processing, context window management, and billing operates at the token level.

The relationship between tokens and words varies by language and tokenizer. In English, common words often correspond to a single token, while less common or longer words may be split into multiple subword tokens. For example, "understanding" might be tokenized as "under" + "standing" or as a single token depending on the tokenizer. Punctuation, spaces, and special characters are also represented as tokens. As a rough approximation, one token corresponds to roughly three-quarters of a word in English, or about four characters.

Tokens have direct practical implications across several dimensions. Context window limits define how many tokens a model can process in a single request, including both input and output. Current context windows range from 4,096 tokens for smaller models to over 1,000,000 tokens for the largest. API pricing is typically calculated per token (often per million tokens), making token efficiency directly relevant to operating costs. Token budgets must be managed carefully in RAG systems, agent workflows, and long conversations to fit relevant context within the available window.

Understanding token economics is essential for AI application design. Developers use tokenizer libraries (tiktoken for OpenAI models, tokenizers for Hugging Face models) to count tokens accurately and estimate costs before deployment. Token-efficient prompting techniques, strategic context truncation, conversation summarization, and caching strategies help optimize the balance between AI system capability and operating cost at scale.