Chain of Thought

What asking a model to think step by step was for

The original technique was a prompt suffix. Add "think step by step", and instead of jumping to an answer the model writes out intermediate steps first and gets more of them right.

The mechanism is worth holding onto, because it explains everything that followed. A model produces one token at a time, each conditioned on what came before. Asking for the working makes those steps part of the output, which gives the model somewhere to do the work rather than having to arrive at a conclusion in a single leap.

That is why it helped most on arithmetic, multi-step logic and anything where the answer depends on an ordered chain of facts, and barely at all on recall or formatting.

What changed when models started reasoning by default

Current models from the major providers reason without being asked. The steps happen, they are billed as output tokens, and depending on the provider they are returned in a separate block or summarized.

So the instruction has moved from a default technique to a fallback. Anthropic's guidance now frames it that way: "Manual chain-of-thought (CoT) prompting as a fallback. When thinking is off, you can still encourage step-by-step reasoning by asking Claude to think through the problem."

The same guidance discourages the hand-written version of it: "Prefer general instructions over prescriptive steps. A prompt like 'think thoroughly' often produces better reasoning than a hand-written step-by-step plan."

When the instruction is now redundant, and when it costs you

There is a documented case where an instruction that used to improve answers now makes them worse, on the same vendor's newer model.

On self-checking, the guidance reads: "Claude Opus 5 is the exception: it verifies its own work well without explicit instruction, and verification instructions carried over from prompts tuned for earlier models can cause over-verification, adding tokens and latency. When migrating to Claude Opus 5, remove these instructions rather than rewriting them."

Read that last sentence carefully. The advice is not to soften the instruction. It is to delete it.

This is the general shape of the problem. Prompts accumulate instructions that were good advice against a model nobody is running any more, and nothing in a prompt carries a date. When you migrate models, the prompt is part of what migrates, and regression testing is how you find out which of those instructions is still earning its place.

A rough sorting, for current models:

• Reasoning is on. Do not ask for step-by-step working. You are paying for it twice, once in the reasoning block and once in the visible output.
• Reasoning is off or unavailable. The instruction still helps, and general wording beats a prescribed procedure.
• A small or older model. The original technique applies as it always did.

All of the above is one vendor's guidance about its own models, and the specifics differ by provider. The habit that transfers is checking the current guidance for the model you are on rather than carrying a prompt forward.

Keeping the reasoning out of what your code parses

Whichever route produces the steps, your code usually wants only the conclusion.

The documented approach is explicit separation: "Use structured tags like <thinking> and <answer> to cleanly separate reasoning from the final output." Ask for the answer inside a tag, and parse that tag rather than the whole response.

Parse defensively even so. On at least one current model with thinking disabled, "the model can occasionally emit internal XML tags into its visible output", which means a parser keyed on a tag can find one it did not expect. For anything your code depends on, structured output gives a schema the response has to satisfy, which is a stronger guarantee than a tag convention.

One more reason to keep them apart: reasoning tokens are billed and they are not the product. If you are showing a user the answer, you are paying to generate text nobody reads. That is often worth it for accuracy, and it should be a decision rather than an accident.

Further reading

• Anthropic, Prompting best practices: the fallback framing, the over-verification warning, and the tag separation.
• OpenAI GPT and o-series: reasoning effort as a parameter rather than a prompt.
• Structured output: parsing the answer without relying on a tag.
• Regression testing: finding the instructions a model migration made redundant.