Self-hosted LLM inference versus cloud APIs

The finding, first

Most write-ups of local AI ask "which model is best." That question does not produce a plan. The question that produces a plan is which hardware tier does each workload need, because that is a capacity question, and capacity planning is what an evaluation is supposed to output.

I run a two-tier fleet and score 5 to 6 local models against a fixed prompt set on it. Three workload classes separated cleanly, and they separated by tier, not by model:

• Embedding and retrieval run fine on a 6GB laptop GPU. They are small, cheap, and disproportionately valuable.
• Interactive assistance wants the 24GB node.
• Long-horizon reasoning wants the 24GB node and patience.

The second finding is narrower and more useful than it sounds: an external GPU over Thunderbolt or OCuLink does not cripple inference. It costs you model load time, not token generation. I explain why below, because the assumption that it does keeps people from buying the cheapest viable on-premises capacity there is.

1. The decision, and who it is for

The decision is whether to buy inference per token from a commercial API, build it on hardware you own, or route between them.

This memo is written for organizations holding data that cannot leave their perimeter: records governed by statute, client intellectual property under contract, source code under NDA. If your data can legally and contractually go to a third party, your evaluation is a normal vendor selection and the weights below are wrong for you. Change them. That is the point of publishing them.

2. Criteria, defined before anything is scored

Weights were set before I scored a single option, from one premise: the reader cannot send this data out. Weights are the reader's to change, and Section 5 shows what happens when you do.

Criterion 8 is two-directional and people usually score it wrong. Cloud gets better for free and also changes without asking, deprecates models you validated against, and silently invalidates your prompt tuning. Self-hosted weights are frozen, reproducible, and get better only when you do the work.

3. Options under evaluation

A. Self-hosted open-weight models on owned hardware. Ollama serving on machines you control, no egress.

B. Commercial cloud API. Per-token or per-seat, frontier capability, someone else's operations team.

C. Hybrid with an explicit escalation path. Local by default; escalation to a commercial model behind a deliberate override, with secret redaction at the model boundary and an audit record of every escalation. The load-bearing word is explicit. A silent fallback to cloud when the local node is busy destroys the entire premise of Option A, quietly, at the worst possible moment.

4. Method

The lab is a single-operator platform I built and run. Its benchmark service loads 5 to 6 local models on demand and scores them on a fixed prompt set covering code reasoning, summarization, extraction, and retrieval-augmented answering. Models are loaded on demand rather than held resident. That trades first-request latency for breadth: you can serve a menu of models on one card, but the first call after a swap pays a load penalty.

The fleet is two tiers on purpose:

• Node A (workstation): RTX 3090, 24GB VRAM, in an external enclosure.
• Node B (laptop): NVIDIA A1000, 6GB VRAM, plus Intel integrated graphics.

A used 3090 at 24GB remains the canonical price-per-VRAM point for self-hosted inference, which makes the capital figures below recognizable to anyone who has priced this.

What each tier can actually serve

Budget roughly 0.6GB of VRAM per billion parameters at Q4 quantization, then add KV cache, which grows with context length.

Honest observations: smaller quantized models are fast enough for interactive work. Larger models are slower but noticeably better at multi-step reasoning, which makes them right for long-running background tasks and wrong for anything a human waits on. A 70B on 24GB requires quantization aggressive enough that the quality you were buying is the thing you destroy.

The external GPU question

An eGPU is widely assumed to cripple inference. It largely does not, and the reason is structural. During autoregressive decoding, the model reads its entire weight set out of VRAM for every token generated. On a 3090 that path runs at roughly 900GB/s. The host bus carries the prompt in and the tokens out, which is kilobytes. Thunderbolt delivers about 2.5-3GB/s in practice and OCuLink about 8GB/s, but neither is in the hot path once the weights are resident.

Where it does hurt is loading. A 32B at Q4 is about 19-20GB; pushing that across Thunderbolt is roughly 7 to 8 seconds, across OCuLink 2 to 3 seconds, against well under a second on a full x16 slot (assuming NVMe read speed is not the real bound, which it often is). With on-demand loading, that is a per-swap cost, not a per-token cost. It also means multi-GPU tensor parallelism, which does hammer the interconnect, is the case where an external link genuinely falls apart.

Limits of this method

Single operator, single-tenant load. I did not stress concurrency at N simultaneous users, and concurrency is exactly where a 24GB node stops being cheap. Quality scoring includes subjective judgment on reasoning tasks. Treat the scores below as a structured argument, not a measurement.

5. Scoring

Scores are 1 to 5, higher is better. Weighted total is the sum of (weight x score), out of a maximum of 500. Every number here is recomputable from this page.

Hybrid pays for its win with the worst operational burden on the board: you run the fleet and manage a vendor and maintain the routing and redaction layer between them. It is the highest score and the most work.

Sensitivity

Each scenario states every weight that changed. All three sets still sum to 100.

Read these honestly. In A and B the spread is 25 points on a 500-point scale, which is 5%, which is inside the noise of a subjective 1-to-5 rating. A 5% gap is not a decision. When scenarios land that close, pick the simpler operation, which is the cloud API. In C, self-hosted and hybrid tie exactly, and hybrid's entire remaining value is the escape hatch.

6. Total cost of ownership

The two options have different cost shapes, and comparing them at one point in time is the classic error. Cloud is operating expense that scales with usage and never ends. Self-hosted is capital, plus electricity, plus the line almost everyone omits: someone's hours.

The operator line is 83% of the cost. Hardware is a rounding error next to a human being.

Break-even. At $200/month per seat, self-hosting pays for itself at 2.42 seats, so call it 3. But that number is a trap on its own, because a single 24GB node saturates at roughly the same point: past a few concurrent interactive sessions, queue depth becomes latency. Cost per seat does not keep falling smoothly. It steps at every node. The honest curve is that a marginal node costs $833 amortized, plus $175 power, plus about $1,200 of incremental operator time (operator hours grow sublinearly), so $2,208 per year for another ~3 seats, or about $736 per seat per year against $2,400. Self-hosting wins on cost at scale. It does not win at 1 to 2 seats, and it never wins on capability.

I pay about $200/month for a commercial coding assistant and consider it money well spent. That is the same math coming out the other way: one seat, spiky usage, frontier ceiling, zero operator hours. The honest conclusion of this evaluation is not "self-host everything."

Variables that move break-even: electricity rate, fully loaded operator cost, used GPU street price, vendor seat and per-token pricing, and utilization. Idle capital is what kills self-hosted TCO. A node at 10% duty cycle costs the same as one at 90%.

7. Where self-hosting loses

• Capability ceiling. On 24GB you cannot buy your way to frontier reasoning. No quantization trick fixes this.
• Spiky demand. Cloud absorbs a 10x Tuesday for 10x of a small number. Your node absorbs it as a queue.
• Bus factor. When the person who maintains the fleet leaves, the platform's real cost doubles overnight.
• Cold start. On-demand loading means the first request after a model swap is slow. Users notice.
• Capability drift. Frozen weights are reproducible and also frozen. Staying current is a recurring project, not a subscription.
• Moving from a desk to a rack changes the shape again: power and cooling budget per rack unit, physical security, remote hands, and a hardware refresh cycle. Consumer GPUs in a colocation cage are a thermal and support conversation, not a purchase order.

8. Recommendation, conditional

• Self-host when the data is barred from third-party processing by statute or contract, the workload is embedding, retrieval, classification, or summarization at steady volume, and a named person owns the system. Retrieval workloads are the strongest case in the entire evaluation: an embedding model is small, cheap, runs on the 6GB tier, and delivers most of the practical value people attribute to the large model.
• Buy when volume is low or spiky, you need the capability ceiling, or nobody's job description contains this system. One to two seats is a buy, essentially always.
• Hybrid when the sensitive majority of your work is routine and a small minority needs the ceiling. Non-negotiable conditions: secret redaction at the model boundary, escalation behind an explicit and refusable override rather than an automatic fallback, and an audit record of every crossing. Without those three, hybrid is just cloud with extra steps and a false sense of residency.

9. What would change this

1. 1.Open-weight capability in the 24-32GB class. When a 32B at Q4 matches the current frontier on your own eval set, criterion 4 moves from 2 to 4 and the base recommendation flips to self-host. Test this yourself, on your tasks. Public benchmarks will not tell you.
2. 2.Price per GB of VRAM. The used 3090 is today's reference point. A 32GB or 48GB card at comparable dollars per gigabyte changes the capital line and the model menu simultaneously.
3. 3.Vendor contract terms, specifically whether you can get no-training and short-retention commitments in writing. That is the cheapest way to raise cloud's residency score, and it costs procurement time rather than capital.
4. 4.Model deprecation cadence. Every forced migration is unbudgeted engineering. Track how often your vendor has retired a model you depended on.
5. 5.Operator turnover. This is the leading indicator most organizations fail to watch, and it is the one that actually decides whether self-hosting worked.