Ollama

What the tool removes from running open weights

Running an open model yourself normally means choosing a runtime, finding the weights, picking a quantization, converting a format, and configuring a server. Ollama collapses that into one command.

ollama run llama3.3

That pulls a packaged model and starts serving it locally. The documentation's framing is that it lets you "use open models in your desktop apps and coding agents, or build them into your application", locally or against its cloud models for machines without a GPU.

The value is in what you no longer have to decide. The cost is that some of those decisions were still made, but not by you.

What a model tag selects

llama3.3 is a tag, and a tag identifies a specific packaged build: a model, at a size, at a quantization, in a particular format.

This is the part worth slowing down on, because the default tag is a choice somebody made about the quality and memory trade, and it is easy to run for months without noticing you made it. A heavily quantized build is smaller and faster and answers differently from the full-precision weights, and the difference shows up first on the hardest cases rather than on the ones you check by hand.

The documentation pages I read do not state which quantization a bare tag resolves to, so no default is asserted here. What to do about it: read the tag you are pulling rather than accepting the short form, and check the model library listing for what the variants are. If you compare a local model against a hosted one and the local one seems worse, the quantization is the first thing to rule out, not the model family.

Pointing existing code at it

Ollama exposes its own API and a compatible surface. The documentation directs you to "use Ollama's API or compatible OpenAI and Anthropic clients", which means code written against a hosted provider can usually be pointed at a local model by changing a base URL and a model name.

const client = new OpenAI({ baseURL: 'http://localhost:11434/v1', apiKey: 'ollama' })

That is the same trick Hugging Face Inference SDK and OpenRouter use from different directions, and it makes a useful workflow available: run your evaluation set against a local model and a hosted one with the same code, and compare.

Expect the compatibility to be good rather than complete. Provider-specific features do not survive a common interface, which is the general point OpenRouter makes about routing and applies here too.

Where a laptop stops and serving begins

The tool is built for one machine serving one person at a time. That is the right design for development, and it is the thing to be clear-eyed about before it appears in an architecture diagram.

Production serving has requirements a local runner is not trying to meet: concurrent requests sharing GPU memory, batching, queueing when the cache fills, and predictable behaviour when more requests arrive than fit. Self-hosted models covers what that costs and why the KV cache is the number that binds.

None of that makes the local tool wrong. It makes it a different tool. The sensible pattern is to use it for development, evaluation and anything running on a single workstation, and to treat "we already run it locally" as no evidence at all about how it will behave serving traffic.

Further reading

• Ollama, documentation: running a model, the API, and the compatible client surfaces.
• Self-hosted models: GPU memory, the KV cache, and what serving requires.
• Meta Llama: one family you can run this way, and its licence.
• Hugging Face Inference SDK: the other client that spans local and hosted.