Choosing the Right Model

Turning "which model" into a comparison you can run

"Which model should I use" is not answerable as asked, and the reason is worth stating plainly: it is missing the task.

Rewritten as something answerable, it becomes: which of these three models produces acceptable output on my task, within my latency budget, at a cost I can pay? That question has an answer, it changes when your task changes, and you can run the comparison yourself in an afternoon.

Six axes for comparing models

Vendors publish comparison tables, and the columns they choose are a reasonable starting list. Anthropic's compares on description, comparative latency, price per million input and output tokens, context window, maximum output, thinking mode, knowledge cutoff, and retirement date.

Six of those are worth carrying into your own comparison.

Capability on your task. Not on a benchmark. The next section is about this, because it is the axis everyone gets wrong.

Cost per request. Not price per token. Also its own section below.

Latency. For streamed responses, measure how long a reader waits for the first visible output. If they need the complete result before continuing, measure total response time. For an overnight batch job, measure whether it finishes inside the processing window. Inference separates these.

Context window. Large enough that this rarely blocks anymore. Current frontier models document 1M tokens. Check it anyway if you are sending documents.

Maximum output. The one people forget. A model with an enormous input window can still cap generation, and a task producing long documents hits that ceiling in production having never hit it in testing.

Availability where you need it. Hosted, in your cloud, in your VPC, or on your own hardware, self-hosted models covers that end. This is the axis most likely to eliminate a candidate outright rather than scoring it lower.

Work down the list and you will usually find one axis eliminating most of the field before quality enters the conversation. That is a good outcome. Comparing two models is much easier than comparing nine.

Reading model pricing correctly

Published prices are per million tokens, and two things about how they are structured make headline comparison misleading.

Input and output are priced differently, often by a lot. Claude Opus 5 is listed at $5 per million input tokens and $25 per million output tokens (checked 2026-09-16), so each output token costs five times an input token. So the same price list produces very different bills for different features. Summarizing long documents is input-heavy; generating long drafts from a short brief is output-heavy. Two features with identical request counts can land at very different bills, and a comparison done on a blended figure will pick wrong for at least one of them.

Discounts are structural, not negotiated. "Batch API requests are 50% off; prompt cache reads cost 10% of the base input price." If your workload tolerates delay, half the list price is available by changing an endpoint. If your prompts share a long prefix such as a system prompt, a document, or a set of examples, prompt caching bills a cache read at 10% of the base input price on Anthropic's published rates (checked 2026-09-16), so the reused portion costs a tenth of what it otherwise would.

So compute cost per request, for your actual request shape, at your actual volume. Take a representative prompt, count its input tokens and a typical output, apply each candidate's two prices, multiply by monthly volume. It takes ten minutes and it regularly reverses the ranking that headline prices suggest. Stop estimating tokens covers counting them properly, which matters here because character-based estimates are wrong by enough to change the decision.

Twenty cases beat a leaderboard

Leaderboards measure someone else's task. That is not a criticism of the methodology; it is what aggregate benchmarks are.

Sentence Transformers puts it well, about embedding models, with its own emphasis: "models that perform well on the leaderboard do not necessarily do well on your tasks, it is crucial to experiment with various promising models." That is from a different corner of the field, and the reasoning transfers. A benchmark averages across domains. Your application is one domain, with its own vocabulary, formats and edge cases, and the model ranked fourth overall may be first on text like yours.

What replaces it is smaller than people expect. Twenty real inputs, with the output you would accept for each.

Take them from real traffic if you have it, or from the examples in the ticket that prompted the feature. Include the awkward ones: the ambiguous request, the one in another language, the one where the right answer is "I don't know." Those separate models; the easy cases do not, since every candidate passes them.

Then run all candidates over all twenty and read the outputs yourself. At this size, reading is the right method and building a scoring harness is premature. LLM evaluations and deterministic evals cover how to scale this into something automated, which is worth doing once the feature is real.

Twenty cases take an afternoon and answer a question no published ranking can: whether a model is good enough at the task you are shipping.

Why a published ranking will not answer this

Model names, prices, context windows and latency tiers all change within months, and a ranking carries no date a reader can check. A list that was accurate when it was written reads exactly the same once it is not.

A ranking also assumes a task. The strongest available model is the wrong choice for a classification job a small model handles at a fraction of the cost, and the cheapest is the wrong choice for a reasoning task it fails. Neither is visible from a leaderboard position.

For current numbers, read the vendor's own comparison page, which carries a date and is maintained. For the choice itself, run the twenty cases.

Keeping the model choice reversible

One design decision makes the rest of this cheap: do not let the model name spread through your codebase.

Put the provider call behind one interface, keep the model identifier in configuration, and keep your twenty cases in the repository. Then re-running the comparison is running a script, and switching is editing a value.

That matters because this decision does not stay made. Models are released, prices fall, and, the one people do not plan for, models retire. Every model ID is "a pinned snapshot" with a published retirement date, and one current model's commitment is "Not sooner than October 15, 2026". A feature built on a specific model has a migration in its future whether or not anyone scheduled it. When that lands, the team with twenty saved cases runs them; the team without makes a guess and ships it.

OpenRouter covers making the switch itself cheap, which is what keeps this decision revisitable rather than made once.

Further reading

• Anthropic, Models overview: a current comparison table, the pricing structure, and the retirement commitments quoted above. Dated, and maintained.
• LLM evaluations: turning twenty cases into a real suite.
• Stop estimating tokens: counting the tokens your cost comparison depends on.
• OpenRouter: making the choice cheap to revisit.