Module 2 · AI engineering foundations · Chapter 7 · Free
Tokens and context windows
Models read tokens, not words
A language model never sees your text as letters or words. A tokenizer first chops the text into tokens: chunks that can be a whole common word, part of a rare word, a number fragment or a punctuation mark. Everything that matters in practice (price, speed, length limits) is counted in tokens.
Type or paste something below. Notice how a long unusual word breaks into pieces and how numbers split up. Real tokenizers are learned from data and differ between models, so treat this as an illustration of the idea, and count with your provider's tool when it matters.
Rules of thumb that hold up reasonably well for English text: a token is often around four characters, and non-English text, code, and numbers usually need more tokens for the same meaning. That is why the same sentence can cost different amounts in different languages, and why messy exports full of IDs and timestamps are expensive to send.
The context window is a budget
The context window is the maximum number of tokens a model can consider in one call. It is shared by everything: your instructions, the conversation so far, any documents you retrieve, the user's question, and the answer the model is about to write. If the sum goes over, something has to give.
Drag the sliders. The numbers are examples, not a specific model, so the window size here is just for the lesson.
The window is a budget, so you manage it deliberately. Common patterns: keep the last few turns verbatim, replace older turns with a short summary, and retrieve only the facts relevant to the current question instead of pasting everything. A larger window delays the problem but raises cost and latency, because every call re-sends the whole history. Chapter 14 (memory systems) goes deeper.
Bigger windows are not free
- Models accept long inputs, and it is tempting to paste everything in "just in case".
- Providers charge per token for input and output, and input is re-sent on every call. A chat that carries 30,000 tokens of history pays for them each turn.
- More input tokens means more work before the first word comes back. For a customer-facing assistant this is felt directly as waiting.
- Research on long contexts (for example the "Lost in the Middle" study) found models can use information in the middle of a long input less reliably than information at the start or end. Behaviour varies by model, so test with your own data.
- Retrieve only relevant passages (Chapter 9), trim or summarize history, and use provider features such as prompt caching for large stable prefixes. Chapter 17 covers latency and cost.
Check your understanding
Recap
- Models read tokens. Price, speed and length limits are all counted in tokens, and tokenizers differ by model.
- The context window is one shared budget for instructions, history, documents, question and answer.
- More context means more cost, more latency and sometimes worse recall. Send less, but better.
Chapter quiz
Four questions. Score 70% or more to mark this chapter complete.