Tokens
Models & ArchitectureThe small pieces of text that AI models read and generate -- like individual puzzle pieces that make up sentences.
Think of tokens like syllables in speech. Just as we break words into syllables to pronounce them, AI breaks text into tokens to process it. The word 'hamburger' might be split into 'ham,' 'bur,' 'ger' -- three tokens.
Tokens are the fundamental units that language models work with. Before an AI can process your message, it breaks it down into tokens -- chunks of text that are usually a word, part of a word, or a punctuation mark. The word "fantastic" might be one token, while "unbelievable" might be split into "un," "believ," and "able" -- three tokens.
Tokens matter to you for two practical reasons. First, pricing: most AI services charge based on the number of tokens you send (input) and receive (output). A longer conversation costs more tokens. Second, context limits: every model has a maximum number of tokens it can handle in one conversation (its context window). If your conversation exceeds that limit, the model starts forgetting earlier parts of the conversation.
A rough rule of thumb is that one token equals about three-quarters of a word in English. So 1,000 tokens is roughly 750 words, and a 128K token context window can hold about 96,000 words -- roughly the length of a novel. Other languages may use more tokens per word, so the same sentence in Japanese or Arabic might cost more tokens than in English.
Understanding tokens helps you use AI tools more efficiently. If you are hitting a context limit, you can shorten your prompts. If you are watching costs, you can estimate how much a task will cost based on the token count. Most AI platforms show you your token usage so you can keep track.
Real-World Examples
- *OpenAI charges per token -- GPT-4 input tokens cost more than GPT-3.5 tokens
- *Claude's 200K token context window can fit roughly an entire book
- *The sentence 'Hello, how are you?' is approximately 6 tokens