Large Language Models (LLMs)

How a large language model predicts text

A large language model predicts what text comes next.

You give it some text, and it produces a continuation one chunk at a time, each chunk chosen from what it judges likely to follow. That single behavior explains most of what you will run into when you build on one. It answers questions because answers follow questions in the text it learned from. It writes code because code follows a description of code. And it invents a refund policy for your company because a plausible-sounding policy is what follows your question, and it has no way to check.

How prompts reach the model as tokens

Your prompt reaches the model as tokens, which are chunks of text. A few characters is typical, and the size varies by word and language. The model reads tokens, produces tokens, and providers bill you per token, so character counts and word counts both mislead you here. The counting rules have their own article.

Everything the model knows about your situation arrives this way. There is no side channel. If your policy, the customer's history, or yesterday's conversation is not in the text you send, the model is working without it.

The context window as a shared input and output budget

The context window is the maximum number of tokens a model can work with at once, and your input and its output draw on the same allowance. A long prompt leaves less room for a long answer.

Window sizes differ by model and change often, so check your provider's documentation rather than memorizing a number. Anthropic publishes current limits per model, as does every other major provider. Plan for the limit to move, and read it from configuration rather than hard-coding it. Then subtract the space you reserve for the answer before you decide how much context fits, or long generations will truncate. What you put in the remaining space is a decision worth making on purpose, which is the subject of context engineering.

How the model generates a response

Tokens, one at a time. That is why streaming exists: the first words can reach a user while the model is still working on the rest. It also explains why long answers take longer than long questions, since reading your prompt happens in one pass while writing the answer happens token by token.

Ask the same question twice and you can get two different answers. The model picks among likely continuations, and that choice involves randomness you can turn down through sampling settings. Turning it down narrows the variation without promising identical text.

Four model behaviors that shape your application code

It does not know your data. Anything private, recent, or specific to your company has to travel in the prompt. Fetching the right documents at request time is retrieval, and it is the standard answer to this problem.

There is a hard ceiling. You cannot send a million-line codebase and ask for a summary. Work that exceeds the window has to be split, summarized, or filtered before it reaches the model.

Models do not read a long context with equal attention. Liu and colleagues measured this in Lost in the Middle: models retrieve information placed at the start or end of a long context more often than the same information buried in the middle. The size of the effect varies by model and task, so treat it as a reason to put important material where the model reads it best rather than as a fixed number. See long context processing.

The output shape is not guaranteed by asking. If your code parses the answer, ask the provider to enforce a schema through structured output rather than relying on the wording of your request.

Which model limits you can fix, and which you design around

You fix the first and the last. Missing knowledge is a retrieval problem with a known solution, and an unreliable output shape is a schema problem with a provider feature behind it.

The middle two are different. The context ceiling and uneven recall across a long context are properties of the model you are calling, so your options are to send less, to send it in a better order, or to break the work into several calls. No prompt wording removes them, and knowing which group a problem falls into tells you whether to reach for engineering or to change what you are asking the system to do.

Further reading

• Vaswani et al., Attention Is All You Need: the architecture behind current models, for readers who want the mechanism.
• Liu et al., Lost in the Middle: the study behind the uneven-recall claim above.
• Anthropic, Context windows: how input, output and the shared limit fit together, with current numbers.