Tokens and Tokenization, Explained

If you build anything with large language models, you run into tokens. API costs are priced in tokens. Context windows are measured in tokens. Rate limits are expressed in tokens per minute. Yet "token" is a fuzzy concept that doesn't map cleanly to words or characters, and that fuzziness has real consequences for how you design and budget applications. This guide covers what tokens are, how tokenization works under the hood, and why the details matter when you're shipping production code.

What a Token Actually Is

A token is a chunk of text that a language model processes as a single unit. Most modern tokenizers split text into subword fragments rather than whole words or individual characters. For English text, one token is roughly equivalent to four characters or about three-quarters of a word — a ratio OpenAI has long cited as a useful approximation. The word "hamburger" might be a single token, while "tokenization" might split into "token" and "ization."

The key point is that tokens are not a universal standard. Each model family has its own tokenizer — the component that converts raw text into token IDs and back again. The same sentence can produce a different token count depending on whether you're using a model from OpenAI, Anthropic, Google, or an open-weight model with its own tokenizer. You can't assume a fixed ratio when estimating costs across providers.

How Tokenization Works

Most production tokenizers use some variant of Byte Pair Encoding (BPE) or a related algorithm like WordPiece or SentencePiece. BPE works by starting with individual bytes or characters, then repeatedly merging the most frequent adjacent pairs into new tokens. The process is trained on a large corpus, producing a fixed vocabulary of token types — typically in the tens to hundreds of thousands. Common word fragments become single tokens; rare or unusual sequences get split into smaller pieces.

This subword approach is a compromise. Whole-word tokenizers would need enormous vocabularies to cover every possible word, including typos, names, and technical terms. Character-level tokenizers avoid that problem but produce long sequences that are expensive to process. Subword tokenization keeps vocabularies manageable while ensuring any text can be represented, even text the model has never seen before.

A practical consequence: tokenization is language-sensitive. English text is generally compact because most tokenizers are trained primarily on English data. Other languages, especially those with non-Latin scripts, often require more tokens per word, which means higher costs and faster context exhaustion. Code also tokenizes differently from prose — whitespace, brackets, and variable names each contribute tokens, and long identifiers can split into multiple pieces.

Why Tokens Drive Cost and Limits

API providers price their models per token, usually with separate rates for input (prompt) tokens and output (completion) tokens. Output tokens typically cost more — sometimes several times more — because generating text requires more computation than reading it. Your application's cost depends not just on how much text you send, but on how much text the model generates, and on which model you use.

Context windows — the maximum number of tokens a model can consider at once — are also token-limited. If your prompt plus any retrieved context plus the model's response exceed the window, the model either truncates earlier content or rejects the request. Rate limits are commonly expressed in tokens per minute, so high-volume applications can hit throughput ceilings even when individual requests are small. Understanding where your tokens go — system prompt, user input, retrieved documents, model output — is the foundation of cost and performance control.

Practical Steps for Builders

Count tokens before sending requests. Most providers offer token-counting endpoints or libraries. OpenAI's tiktoken library and Anthropic's API both let you estimate token usage locally, which is useful for budgeting, deciding whether retrieved context fits, and avoiding rate-limit surprises. Counting characters is a rough proxy but breaks down quickly with non-English text, code, or unusual formatting.

Be deliberate about output length. Because output tokens are the expensive ones, letting a model generate a long response when a short one would do is the fastest way to inflate costs. Set max_tokens or equivalent parameters thoughtfully, and design prompts that ask for concise answers when detail isn't needed. This matters especially in agentic or multi-turn workflows where outputs may be fed back as inputs, compounding token usage across steps.

Watch for tokenization quirks in edge cases. Repeated characters, unusual scripts, base64-encoded data, and certain file formats can tokenize inefficiently — a short base64 string might consume far more tokens than you'd expect. If your application handles arbitrary user input or file uploads, test with realistic edge cases rather than assuming average ratios hold. For builders exploring models, tools, or roles in this space, Neural Goldmine's jobs feed and tool listings include positions where this kind of practical knowledge is assumed rather than taught — understanding tokenization won't fix a bad prompt, but it will help you estimate costs, debug context issues, and reason about why a model behaves differently than expected.