Pre-trained Models

What "pre-trained" is claiming

The word describes a relationship: the model was trained before you got it, by someone else, on text you will never see, at a cost you could not pay.

That is a plain statement of dependency and it is worth sitting with. You are building on a artifact whose contents you cannot inspect, whose behavior you can only measure from outside, and which you did not shape.

Several of the problems you will hit are not bugs in your application. They are consequences of how that artifact was made, and knowing which stage produced a symptom tells you which of your options can address it.

Predicting text is where the capability comes from

Pre-training is one task repeated at enormous scale: given some text, predict what comes next.

No labels, no question-and-answer pairs, no human grading. Take text, hide the next token, ask the model to guess, adjust the weights by how wrong it was. Repeat across a corpus large enough that the only description anyone gives is a fraction of the public internet.

The surprising thing, the finding the whole field rests on, is how much capability falls out of a task that simple. To predict the next token in a paragraph about a legal dispute you need something about how law works. To complete a function you need something about the language and about the intent. Grammar, facts, translation, style, rudimentary reasoning: none was taught explicitly, and all of it emerges as a side effect of getting good at prediction.

Two consequences follow immediately, and they explain a large share of the application problems in the table further down.

Everything the model knows came from that corpus, and the corpus ended on a date. And the model was optimized for plausibility, not truth. A fluent, well-formed, confident wrong answer is not a malfunction. It is the objective working correctly.

Why the raw model would not answer your question

A model straight out of pre-training is not something you would want to call.

Ask it "What is the capital of France?" and the honest continuation of that text might be another quiz question, because in its training data that string appears in lists of quiz questions as often as it appears followed by an answer. It completes documents. It does not answer people.

Turning that into an assistant happens afterward, in stages usually grouped as post-training. First a model is shown examples of instructions followed well, so that a request reads as something to satisfy rather than continue. Then it is tuned against human preferences between candidate responses, which is where tone, refusals, formatting habits and hedging come from.

I want to be careful about how firmly to state this. Labs describe their post-training at different levels of detail and do not use consistent vocabulary for the stages, and no first-party page I read lays out a canonical named sequence. Treat the two-stage description above as the widely used industry framing, accurate in shape, not as a specification any particular model followed.

What matters for your application is the division, and it is reliable: the knowledge is from pre-training, and the manners are from post-training. When a model refuses something reasonable, answers in a format you did not ask for, or opens with a paragraph of preamble, you are looking at post-training. When it states a fact that is wrong or stale, you are looking at pre-training.

Two cutoff dates, and the gradient between them

Everyone has heard of the knowledge cutoff. Fewer people know there are two, and the gap between them explains a failure that otherwise looks random.

Anthropic's models table lists "Reliable knowledge cutoff" and "Training data cutoff" as separate rows, and for some models they differ by months. One current model shows a reliable knowledge cutoff of February 2025 against a training data cutoff of July 2025. The page defines the first as "The date through which the model's knowledge is most extensive and reliable" and the second as "the broader range of data used."

So the boundary is not a wall. It is a gradient.

Something that happened three years ago has been written about repeatedly, in news, analysis, forum arguments, and summaries of summaries. Something that happened a month before training stopped was written about once or twice, by whoever was first, and nothing has confirmed it. Both are in the corpus. Only one is in it thickly.

That produces a specific, nasty failure mode. In the window between the two dates, the model has seen something. It will answer. It will sound as confident as it does about the well-established material, because confidence tracks fluency rather than evidence. That window is where a model is most likely to be wrong in a way you will not catch.

The practical response: for anything recent, do not rely on the model's own knowledge even when it clearly has some.

Matching a symptom to the stage that caused it

Diagnose the stage and your options narrow to the ones that can reach it.

The middle column groups those symptoms into three causes. Knowledge problems are context problems. The model cannot learn your refund policy by being asked nicely, and it cannot learn yesterday's news at all, but it can read either if you put it in the prompt. That is the whole argument for retrieval, and it is why a knowledge gap is usually an easier problem than a behavior gap rather than a harder one.

What you can still change about a pre-trained model

Your options divide by which stage they reach.

Prompting changes nothing about the model and everything about what it sees. It reaches post-training behavior well, because you are working with the instruction-following the model was tuned for, and it reaches knowledge gaps whenever you can supply the knowledge, which is what providing context is about.

Retrieval is prompting at scale: find the relevant text at request time and put it in the window. It is the correct answer to every knowledge problem in that table, and it works because it does not need to change the model.

Fine-tuning adjusts the weights, which sounds like it reaches further than it does. It is effective at behavior: format, tone, a consistent structure, a domain's conventions. It is a poor way to teach facts, because facts change and retraining does not, and because the cost of updating a fine-tune is not the cost of updating a document. Fine-tuning is the full argument.

Nothing you can do reaches pre-training. That corpus is fixed, that date has passed, and the capability ceiling it produced is the ceiling. When a task is beyond a model, the answer is a different model, and choosing the right model is how to find out which.

Knowing which of those four you are in saves the most common wasted effort in this field, which is fine-tuning a model to teach it facts.

Further reading

• Anthropic, Models overview: the two cutoff rows and their definitions, plus context windows and retirement dates per model.
• Large language models: inputs, context and limits for an application developer.
• Fine-tuning: what adjusting the weights can and cannot do.
• RAG use cases: the answer to every knowledge problem in the table above.