Common Terminology

What a model is, and what training produced it

A model is a large set of numbers, called weights, plus the code that runs them. Training is the process that produced those weights, and it happened before you arrived: somebody ran an enormous computation over an enormous corpus, and the result is a file.

Inference is every call you make to that file afterward. When you send a prompt and get text back, you paid for inference, not training.

That split explains most early confusion. The model does not learn from your conversation. Weights do not change when you chat with it, and nothing you send today affects what it knows tomorrow.

Fine-tuning is further training that adjusts weights for your task. It is good at behavior and format, and it is not how you give a model facts. See fine-tuning.

Tokens and the context window

A token is the unit a model reads and bills for, usually a few characters, never reliably a word. Text is split into tokens before the model sees it, and the count varies by language and by content. Stop estimating tokens covers why the count never matches anything you can compute by hand.

The context window is the maximum number of tokens a model can work with at once, and it is shared: your input and its output draw on the same allowance. A long prompt leaves less room for a long answer.

Two things people miss. Images and audio become tokens too, so media competes with text for the same budget. And reasoning tokens, produced by models that think before answering, are billed as output and are not visible to you.

What a call costs and how it fails

Providers bill per token, input and output priced differently, with output usually the more expensive of the two. A retry costs the same as the original call. A conversation costs more each turn, because the history is resent every time.

Two terms for failure. Truncation is generation stopping because it hit an output limit, and it arrives as a successful response, so you check the field reporting why generation stopped rather than assuming a short answer was short. Rate limits cap both requests and tokens per period, so many small calls and one enormous call can each trip a limit.

Inference covers the lifecycle in full.

What goes into a prompt

A prompt is everything you send. It is not only your instruction; it is the system instruction, the conversation so far, any documents you attached, and the tool definitions you declared.

Retrieval is fetching relevant documents at request time and putting them in the prompt, which is the standard answer to a model not knowing your data. Grounding is tying the answer to those supplied sources, and telling the model to say when they do not contain the answer. RAG, retrieval-augmented generation, is the pattern combining the two.

Context engineering is the discipline of deciding what occupies the window, as against prompt engineering, which is the wording of the instruction.

A hallucination is a confident, fluent, wrong answer. The word makes it sound exotic; the mechanism is ordinary. The model produces a likely continuation, and when it has no grounds for the fact, a plausible-looking one is still what follows.

What a system adds around the model

Sampling parameters, chiefly temperature and top-p, control how a token is drawn from the model's predicted distribution. They change variation, not knowledge, and temperature 0 is not a reproducibility guarantee. See sampling parameters.

Structured output constrains the response to a schema, so your code can parse it. Function calling, also called tool use, is different: the model returns a request for an action, and your code decides whether to run it. Structured output shapes the answer; function calling requests an action.

An agent is a system that loops: the model chooses a tool, your code runs it, the result goes back, and it repeats until the task is done. The loop and the tools make it an agent, not the model.

Evaluation is measuring quality across many examples rather than asserting one output, and it is how you tell whether any change helped.

Four confusions worth clearing up

Training against inference. Almost everything you do is inference. If someone says the model "learned" from a conversation, they mean the conversation was in the prompt.

Fine-tuning against retrieval. "Fine-tune it on our documentation" is usually the wrong instrument. Fine-tuning shapes behavior; retrieval supplies facts.

Agent as anything that calls a model. A single call with a clever prompt is not an agent. Without a loop and tools, it is a model call.

Temperature as a correctness knob. Lowering it makes output more repetitive, not more accurate. A wrong answer at temperature 0 is a wrong answer you now get consistently.

Further reading

• What is an AI engineer: these terms in the context of one worked feature.
• Large language models: where predictions, tokens and the window come from.
• Inference: the request lifecycle, cost model, and failure modes.