Why does self-hosting sometimes lose to an API even when the per-token cost is lower?

Because you rent the GPU by the hour whether or not it is busy. The per-token figure here assumes 100 percent utilisation. If your endpoint is idle half the day, your real cost per token doubles. APIs charge only for tokens actually used, so they win for low or bursty volume. Self-hosting wins above a break-even request rate.

What batching efficiency does this assume?

About four times the single-stream rate, which is conservative for a well-tuned vLLM or TensorRT-LLM endpoint under steady load. A busy server with long shared prefixes can do much better; a lightly loaded one does worse.

Are the prices accurate?

They are rough mid-2026 on-demand rates across the major clouds and GPU marketplaces, meant for a first comparison. Spot and committed-use pricing is much lower. Always plug in a real quote and the API price you are actually comparing against.

What about the cost of the fine-tune itself, or storage and egress?

Not included. This compares steady-state inference serving only. A one-off fine-tune is a separate, usually small, capital cost; storage and egress are provider-specific line items.

// cost

Cloud GPU vs Local — LLM Cost Calculator

The smallest cloud GPU tier that fits your model, its dollars per million tokens, and how that compares to an API.

Model

QuantizationContexttokensAPI price to compare$/1M out tok

Smallest tier that fits

L40S 48GB

$1.60/hr on-demand

Self-host cost

7.307$/1M tok

vs API

0.6$/1M tok

$1.60/hr ÷ 3600 ÷ (15 tok/s × 4 batched) × 1e6 = $7.307 / 1M tokens

The API is cheaper here. Self-hosting a rented GPU only beats it at high, steady utilisation.

Prices are rough mid-2026 on-demand figures across the major clouds and GPU marketplaces — edit the API field and treat the tier rate as a placeholder for a real quote. Presets verified 2026-09. Assumes ~4× batching efficiency and 100% utilisation; a bursty workload changes the maths entirely.

notes

What it does

Takes your model, quantization and context, and finds the smallest single GPU tier whose memory fits the workload.
Estimates that tier’s decode throughput, applies a batching factor, and divides the hourly rate by tokens per hour to get dollars per million tokens at full utilisation.
Puts that next to an API price you type in, so you can see which way the comparison points.

The break-even idea

A rented GPU costs the same whether it serves a million tokens an hour or none. So the honest comparison is:

self-host $/token  =  hourly_rate  ÷  (tokens_per_second × 3600 × utilisation)

At 100% utilisation self-hosting is usually cheaper per token than a frontier API. But your utilisation is rarely 100%. Work out your real request volume, divide the monthly GPU rental by the tokens you actually generate, and compare that to the API bill.

Worked example

Llama 3.3 70B at Q4_K_M, 8k context ≈ 42 GB → smallest tier that fits is an A100 80GB at ~$2.40/hr.

Decode ≈ 40 tok/s single-stream, ×4 batched ≈ 160 tok/s.
$2.40 ÷ 3600 ÷ 160 × 1e6 ≈ $4.20 per million output tokens at full load.
If a comparable API is $0.60/1M, the API wins unless you are running that GPU near-continuously — roughly 400M+ tokens a month before self-hosting breaks even, and that is before you count ops time.

Notes

Prices are placeholders — get a real quote. Presets carry a verified date. This is steady-state inference only; training cost, storage and egress are separate.

faq

Why does self-hosting sometimes lose to an API even when the per-token cost is lower?: Because you rent the GPU by the hour whether or not it is busy. The per-token figure here assumes 100 percent utilisation. If your endpoint is idle half the day, your real cost per token doubles. APIs charge only for tokens actually used, so they win for low or bursty volume. Self-hosting wins above a break-even request rate.
What batching efficiency does this assume?: About four times the single-stream rate, which is conservative for a well-tuned vLLM or TensorRT-LLM endpoint under steady load. A busy server with long shared prefixes can do much better; a lightly loaded one does worse.
Are the prices accurate?: They are rough mid-2026 on-demand rates across the major clouds and GPU marketplaces, meant for a first comparison. Spot and committed-use pricing is much lower. Always plug in a real quote and the API price you are actually comparing against.
What about the cost of the fine-tune itself, or storage and egress?: Not included. This compares steady-state inference serving only. A one-off fine-tune is a separate, usually small, capital cost; storage and egress are provider-specific line items.