AI glossary Models & Architecture
What is a context window in an LLM?
← All glossary termsThe maximum amount of text, measured in tokens, that a language model can take into account in a single request, including its own reply.
Explained
The context window is the maximum amount of input and output, measured in tokens, that a large language model can process in one request. Everything the model sees counts against it: system instructions, conversation history, retrieved documents, tool results and the answer it writes. Larger windows allow longer documents and conversations, but they raise cost and latency, and models do not always use information in the middle of a long input as well as information at the start or end. Choosing what goes into the window is therefore as important as its size.
How the context window works
Models read text as tokens, pieces of words whose length depends on the language and the tokeniser. Each request sends the model a sequence of tokens, and the window is the ceiling on how long that sequence, plus the generated reply, can be.
The model has no memory between requests beyond what is sent in the window. In a chat, the history is resent each turn; in an agent, tool results pile up step by step. When the limit is reached, something must be dropped, summarised or moved out to retrieval.
Bigger is not always better
- Cost and latency usually grow with the number of tokens sent.
- Research has shown that models can use information at the start and end of a long input better than information in the middle.
- Irrelevant text in the window can distract the model and lower answer quality.
- A large window does not replace access control: anything placed in it can appear in the answer.
Working within the limits
The usual techniques are retrieval-augmented generation (RAG), which brings in only the relevant passages; summarising older conversation turns; trimming tool outputs to what matters; and keeping stable instructions short. Taken together, deciding what the model sees at each step is what is now called context engineering.
Frequently asked questions
What is a token in a context window?
A token is the unit of text a model reads: often a word fragment, a whole short word or a punctuation mark. The number of tokens for the same text varies by language and tokeniser. Context window sizes and API prices are both expressed in tokens.
Does the context window include the model's answer?
Yes, in most models the window covers both the input and the generated output. Providers also set a separate maximum for output tokens. If the input is close to the limit, there is less room for the reply.
What happens when you exceed the context window?
The API returns an error, or the application drops or summarises earlier content to make room. In chat products this can look as if the assistant has forgotten the start of the conversation. Well-designed systems manage this explicitly rather than letting text fall off silently.
Does a large context window replace RAG?
Not in most enterprise cases. Sending everything is costly and slow, permissions still need to be enforced per user, and quality can drop with very long inputs. Long windows and RAG work well together: retrieval selects, the window holds what was selected.
How big are context windows today?
It depends on the model and changes with each release. Current window sizes are published in each provider's documentation, which is the reference to check before designing a system around a limit.
Decision guides that use this term
Where it shows up at Thinkia
Further reading
Related terms
Sources
- Anthropic · Context windows (Claude documentation)
- Liu et al. · Lost in the Middle: How Language Models Use Long Contexts
- Google AI for Developers · Long context (Gemini API documentation)
Updated: 2026-10-21