Self-Hosted Models

What you take on when you stop paying per token

Calling a hosted API, you send a request and someone else owns capacity, availability, and the hardware. Running open weights yourself, you own all three.

The work that arrives is familiar infrastructure work: sizing hardware, planning for peak load, deciding what happens when demand exceeds what you provisioned, keeping it running during an upgrade. None of it is exotic. It is all work that was previously in your bill rather than on your team.

The question is whether what you get back justifies that. Sometimes it does, and the reasons are rarely the ones people lead with.

Where GPU memory goes

Two claims on the same pool, and the second one is the one teams miss.

Model weights are predictable. Parameter count times bytes per parameter, which depends on the precision you load at. This is the number everyone checks.

The KV cache holds the attention keys and values for every token in every active request. vLLM pre-allocates it from whatever memory the weights leave, and the size of that allocation is what a gpu_memory_utilization setting controls. Cache use then "grows linearly with sequence length and batch concurrency."

Read that last phrase carefully. Cache consumption scales with how many conversations are open and how long each has run. Your capacity is not a property of the model; it is a property of your traffic.

This is why a deployment tested with one user at a time falls over in production. The weights fit exactly as predicted. The twenty concurrent conversations, each carrying several thousand tokens of history, did not.

What happens when the KV cache runs out

Requests get preempted. And the documented consequence is the part worth remembering:

Preempted requests are recomputed when sufficient KV cache space becomes available again.

Recomputed, not resumed. Work already done is discarded and done again. So over-subscribing does not trade throughput for latency in a straight line. It wastes compute, which reduces throughput, which lengthens the queue.

The levers point in opposite directions. Raise gpu_memory_utilization or tensor_parallel_size to give the cache more room, and you get closer to running out of memory outright. Lower max_num_seqs or max_num_batched_tokens to reduce pressure, and you serve fewer requests at once. There is no setting that gives you both; you are choosing a point on a curve, and you need your own traffic to choose it.

Quantization as a deliberate trade

Quantization stores weights at lower precision. Hugging Face describes it as lowering "the memory requirements of loading and using a model by storing the weights in a lower precision while trying to preserve as much accuracy as possible."

Weights are typically fp32, with fp16 and bf16 common for large models, and quantization methods go further to int8 or int4. The phrase doing the work is "trying to preserve." Accuracy is what you spend, and the further down you go the more you spend. At the extremes of 1 to 2 bits, some methods need a calibration step to stay usable at all.

The memory you free goes somewhere useful: a larger KV cache, more concurrent requests, or a bigger model on the same card. That can be a good trade. It is still a trade, and the only way to know what it cost is to run your evaluation set before and after.

Comparing the cost models honestly

Per-token billing scales with use. Zero traffic costs nothing, and ten times the traffic costs about ten times as much.

A rented GPU bills for wall-clock time. It costs the same whether you send a million requests through it or none.

So the comparison turns on utilization, not on the price of either. A GPU busy most of the day can be cheaper per request than an API. The same GPU serving bursty traffic with long quiet periods is money spent on idle hardware.

Do the arithmetic with your own numbers: your request volume, your token counts, the instance you would rent. And include what is easy to leave out. The engineer time to run this, the second instance you need for redundancy, and the headroom you must provision for peak all belong in the total. Any figure quoted in an article, including this one, would be wrong for your workload.

Reasons to self-host that have nothing to do with cost

These decide it more often than the arithmetic does.

Data residency. Some data cannot leave your infrastructure, and no discount changes that.

Latency floors. A model on your own network avoids a round trip to a provider and whatever queuing sits behind it. When you need a predictable ceiling rather than a good average, owning the hardware is how you get one.

Model stability. Hosted models get deprecated and retired on the provider's schedule, and behavior shifts under you between versions. Weights on your disk behave the same way next year as they do today. For anything you have validated carefully, that is worth real money.

Modification. Fine-tuning, adapters, and unusual sampling are all yours to do.

If none of these applies and your utilization is low, a hosted API is the straightforward answer. See closed versus open source models for the wider comparison.

Further reading

• vLLM, Optimization and tuning: the memory settings, preemption, and the throughput trade.
• Hugging Face, Quantization: what the methods do and what they cost.
• Inference: the request lifecycle against a hosted API, as the contrast case.