Closed vs Open Source Models

"Open" describes the weights, not the project

The first problem with this comparison is the word. Almost nothing marketed as an open-source model is open source in the sense the term has elsewhere.

What you usually get is open weights: the trained parameters, downloadable, runnable. What you usually do not get is the training data, the training code, or a licence without conditions. Several prominent releases carry acceptable-use terms or scale thresholds that a genuine open-source licence would not have.

So read the licence before the benchmark. Three questions cover most of it: may you use it commercially, may you use its outputs to train another model, and does anything change above a user or revenue threshold. Any of those can rule a model out after you have built on it, which is the expensive way to find out.

What changes

Six things differ, and only two of them are about cost.

The two that decide most real cases are the second and third, not the last.

Version stability is underrated

A closed model can change beneath a working system. An alias resolves somewhere new, behaviour shifts, and the prompt you tuned last quarter performs differently. LLM observability suggests recording the responding model against the requested one for exactly this reason.

Downloaded weights do not move. That is worth real money in a regulated setting, or anywhere you have to demonstrate that a decision made today would be made the same way next year. It is also a liability, because nothing improves unless you do the work.

Data egress is often the whole argument

For a team that cannot send customer data to a third party, the comparison ends here. No benchmark matters, because the closed option is unavailable.

Note that this is often solvable without self-hosting. Enterprise agreements, zero-retention arrangements and in-region deployments exist and are not the default terms. Check what your provider will sign before concluding you need your own GPUs, because self-hosted models sets out how much you are taking on.

Cost only favours open weights at scale

The comparison people expect to be decisive usually is not.

Per-token pricing scales with what you use, including down to nothing. A GPU you have provisioned costs the same whether it serves a million requests or none, so the crossover point sits wherever your utilization makes a fixed cost cheaper than a variable one. Below that line, and most workloads are below it, the API is cheaper as well as easier.

And the honest accounting includes the engineer who keeps it running. Self-hosted models covers the parts that get left out of the spreadsheet: GPU memory, the KV cache, quantization trades and what happens under load.

How to decide without a religious argument

In order:

1. Does the data have to stay put? If yes and your provider will not accommodate it, open weights, and budget for the operations.
2. Do you need a frozen model you can attest to? If yes, open weights.
3. Is the task at the frontier of what models can do? If yes, closed, for now.
4. Is utilization high and steady enough to beat per-token pricing? If yes, run the numbers.
5. Otherwise, closed API, and keep the choice reversible.

That last point does most of the work. Choosing the right model argues for keeping the model swappable, and the same discipline makes this decision low-stakes: an application behind a thin provider interface, with an eval set of its own, can test the other option in an afternoon instead of arguing about it.

Further reading

• Choosing the right model: the six axes, and keeping the choice reversible.
• Self-hosted models: what you take on when you stop paying per token.
• OpenAI-compatible APIs: the interface that makes swapping feasible.
• LLM observability: how you notice a closed model changing under you.