An open AI model has many advantages. Most of them do not require your own hardware.

This is where I am a fan of open models. I can keep a specific version, change the runtime, quantization, or cache, or add my own adapter. I don't have to wait for the provider to expose such an option through its API.

And if the vendor shuts down the API? Either there are plenty of others, or I can take the downloaded weights and spin up a GPU somewhere else.

If the team developing the model shuts down, I retain today's capabilities. That does not automatically give me its future innovations. And switching to a different model is not just changing a URL, either: tool calling, response formats, or quality on my task may behave differently.

None of that requires me to buy a server, though. The same model is offered by large providers as well as smaller and local ones, or I can run inference on my own VM — so we have the option to move elsewhere relatively easily, and I retain the flexibility to change anything at any time (provider, hardware type, region).

On June 12, 2026, Anthropic disabled Fable 5 for all users following a US export directive; it restored global access on July 1. It explains that it was unable to verify users' nationality in real time to apply the restrictions selectively. This was not an ordinary server outage. Downloaded open weights reduce dependence on this kind of off switch for a particular service, but they are not an exemption from laws or export rules.

Although it is hard to hide weights once they are on the internet, access to them does not mean you can use them without restrictions. Companies and governments can contractually prohibit Chinese models in their contracts, and a government can restrict use in healthcare or when working with children, for example. You may end up with a model you can only use for "personal use."

Imagine I am doing an authorized penetration test or analyzing logs from a compromised system. I need to work with offensive techniques too, otherwise I can hardly understand the incident. Or I may be doing research in AI or biology. A filter can see a dangerous topic and stop the work or switch me to a less capable model, which can be limiting.

An open model does not automatically mean a model without guardrails. Refusal behavior can be learned into its weights. The application, license, or infrastructure operator can add further restrictions.

Running inference myself lets me decide where data flows, who sees it, and what gets stored. That is a real advantage, but it does not mean that using a per-token API automatically gives someone access to my data, or that a cloud operator must be able to see inside my VM. An API can have a zero retention policy that limits data retention for supported features and under agreed conditions. Confidential computing, in turn, can protect data in memory even from a privileged infrastructure operator. These are different protections — zero retention alone does not mean the service never processes data during inference.

What differs is how much I configure and take responsibility for myself. But that does not tell us which option is more or less secure; it depends on the details.

And let me return to training data: open weights alone do not reveal its origin and are not proof that the model has no undesirable or covertly conditional behavior (such as a so-called sleeper agent). The location of the server does not resolve this question of trust in the model on its own.

If a suitable region, private access, and the service's contractual terms meet my needs, sensitive data does not automatically force me on-prem.

Running the model myself lets me reserve capacity for my application and tune it for a fast response rather than the maximum number of tokens per day. In other words, I can influence inference parameters and prioritize more tokens per second for one request at the expense of total capacity. Typically, I reduce batch size: an individual request runs faster, but the cost per token can rise significantly.

Commercial providers offer similar choices. For example, Claude fast mode uses the same weights with faster inference settings at a higher price. It is not a different, smarter model. However, Anthropic does not describe the specific implementation as merely reducing batch size, and the speedup mainly concerns token generation, not time to first token.

Besides the network, I also wait in a queue, for input processing, and for response generation. A well-run remote API can beat a small local server. With voice or many short agent steps, on the other hand, I will notice network round trips more than with a single long response.

When I want to control latency, my own cloud VM may be enough. When I have to work without the internet, it isn't. That is where an on-device model or a local server makes sense.

But where a system must not have internet access, I often do not need an LLM at all. An attack drone can use "AI" in the form of computer vision; for that task, it does not need to discuss the differences between the approaches of Plato and Socrates. Offline AI is therefore not automatically an argument for a local large language model.

With a laptop, I would first check whether a smaller model can handle my task at all. It may be excellent for extraction or classification; a complex coding agent is a different assignment. For the coding I expect from an agent, I find anything that can run on a laptop completely unusable. That is my bar for this way of working, not a claim that a small model cannot write a useful function.

I think open models such as GLM 5.3 or Kimi K3 are amazing, but they are not in the laptop league. They are not both the same size, either: the published GLM 5.3 weights have approximately 753 billion parameters, while Kimi K3 has 2.8 trillion. To run them efficiently, I need large multi-GPU systems, and for the largest models a cluster whose price will be in the tens of millions of Czech crowns. The hardware Moonshot AI recommends for running Kimi K3 comes to almost 100 million Czech crowns.

Here I would first separate two kinds of savings. Choosing a cheaper model is not the same as running the same model more cheaply.

For many tasks, I can take a smaller open model with a cheap API instead of an expensive frontier model and save money. If it can do the task, why not? I get the savings immediately, without buying a GPU. But the license alone does not guarantee them: they depend on pricing, quality, the number of attempts, and how much work a human ultimately has to finish. Typically, we talk about the so-called Pareto-optimal frontier: models offering the best price-performance trade-off. For example, the very inexpensive GPT 5.6 Luna is on this frontier — a closed-source model does not necessarily have a poor price-performance ratio. Similarly, current Mistral models are quite far from that frontier — they are open, but their results relative to hardware requirements are not as good as those of others.

The second question is different: once I have chosen a model, is it cheaper to buy its tokens as a service, run it on IaaS, or buy my own hardware?

And that is exactly what I want to calculate in the second part. We have covered control, security, and restrictions. Now I will not justify a server with other advantages — I am interested in the price of the same work.

I tested both Mistral and Nemotron, but from here on I will only discuss NVIDIA Nemotron 3.5 Lightning 30B-A3B in BF16. The comparison principle also applies to other models; the specific speed and break-even thresholds obviously do not carry over.

I used an Azure Standard_NC24ads_A100_v4 VM with one A100 80 GB PCIe and vLLM 0.28.0. Each request had 16 384 input and 512 output tokens. I tried 1, 5, 10, 20, 50, and 100 concurrent requests, each with and without a repeated prefix. Each combination was repeated three times — 36 measurements and 4 017 successful requests in total.

The shared-prefix variant simulates a coding agent repeatedly working in one project: I resend instructions, tools, and part of the context (so this is a test with a large proportion of cached tokens). Without a shared beginning, the model does more work on new input (the uncached scenario).

The A100 is a fairly old card; I did not have another one available — but I think that is fine for an order-of-magnitude comparison on a small model, and the trend we get a feel for here will be similar for larger models and newer cards.

Let's imagine the GPU is serving only me. It loads weights and generates tokens one by one. When it serves more requests together, it can make better use of work over the same weights. Batch size is the size of that jointly processed batch.

Modern serving uses continuous batching: completed requests leave and new ones keep arriving. From the outside, I set client concurrency, not a fixed size for every internal batch. Concurrency 50 does not mean 50 registered users, either, but 50 requests in progress at the same time.

What did that do to economically useful capacity? The tables show aggregate output tok/s for the entire card, not one person's speed. TTFT is time to first token; all figures are averages of three repetitions.

With cache, going from one to fifty streams gave me approximately 5.5 times the total capacity. But an individual's response takes longer. At one hundred, throughput actually falls. So "I'll add a bigger batch and get cheaper" does not hold forever.

To me, this gives us a simple rule for the calculator: I can only count capacity whose latency still meets my needs. An overnight batch and an interactive assistant are different scenarios, and each calls for a different batch size.

Now for the second axis. Concurrency tells me how much work I do at once. Time utilization tells me how long I have that work at all.

One hundred percent of the time at one stream is not the same volume as one hundred percent at fifty. And a reading from nvidia-smi answers neither of these economic questions. What matters to me is completed useful work across which I can spread the costs.

Eight hours, five days a week is only 23.8% of the calendar. And even then I would have to maintain the selected concurrency nonstop during working hours. But people attend meetings, agents wait for tools, and demand fluctuates. Sixty percent utilization means over a hundred hours of work per week. For that, I already need something like multiple time zones, multiple applications, or an overnight task queue.

This is where a large provider has a practical advantage: it combines demand from many customers. When I finish, someone else starts. My own server will not acquire such a queue of work by itself.

Economies of scale can give a provider substantially lower costs: cheaper hardware, a more efficient datacenter, custom kernels or inference chips, or better-tuned solutions (for example, prefill/decode disaggregation or speculative decoding). More on that below.

I compare five years and three ways of running the same model. For the VM, I also distinguish between switching it on only for a batch and holding capacity long-term.

The API rate is the public Fireworks on Foundry reference from a September 3, 2026 snapshot.

Cloud reservation and ownership are not orders of magnitude apart here. The difference comes to USD 12 776.87 over five years, or about USD 2 555 per year. That is how much room is left for additional on-prem costs relative to the VM before the modeled lead disappears.

Let's change one thing at a time: how well I use the time I pay for, and how many requests I process at once. In both scenarios, I use a local shared prefix and assume that the API bills 64% of input tokens as cheaper cached tokens. I pay the regular rate for the remaining input; output is generated and billed separately.

In short: self-hosted inference needs enough work both over time and at once. The API's cached-token discount also affects the result: the cheaper it makes the API, the more work I need for self-hosting to pay off.

Without a shared prefix on either side, at the measured concurrency of 50, the required utilization is 56.18% for the reservation and 44.73% for on-prem. Similar percentages, but different capacity and significantly worse latency: the first token takes 17.72 s.

H = 43 800 hours over five years
u = productive share of the calendar
R = measured output tok/s × 3600 / 512
h = assumed share of input tokens billed by the API as cached
p = [16384 × ((1 − h) × 0.06 + h × 0.01) + 512 × 0.22] / 1 000 000
API = H × u × R × p
PAYG allocated throughout the comparison period = H × USD 4.775
reservation = USD 77 611.67
on-prem = USD 59 316 + H × u × USD 0.126

I want open models among my options. For control, the ability to keep a version, custom behavior, and somewhere to go when a service no longer suits me. But I do not need to buy a server for most of these advantages.

And the money? For this model and workload, a cheap API works out better at low concurrency or with irregular work. When I have enough concurrent work for a large part of the week, self-hosted inference can pay off. Then I would primarily compare a reserved cloud VM with a genuinely complete on-prem budget, not the purchase price of a card with PAYG running forever.

Our working scenario, with 64% of input billed by the API as cached, needs, at concurrency 50, roughly 54% of time for the reservation and 43% for on-prem. Meanwhile, on-prem's modeled lead over the reservation is only about USD 2.6 thousand a year for further differential costs. An operational estimate can turn that around quite easily.

So, to me: first choose a model that can do the work. Then state the control and responsiveness I need. And only then ask who will operate the GPU. Not paying a provider's margin is a pleasant thought. Paying for my own unused server, less so.