Self-Hosted LLM vs API Cost Calculator

Self-Hosted LLM vs API Cost Calculator

Emerging AI Skill PRO

Enter Calculation

GPU deployment

The hourly price is an example. Enter the rate from your cloud quote or your own amortised hardware cost.

Model

Defaults: Llama 3 70B (80 layers, dimension 8,192, 64 heads, 8 key/value heads). Copy the values for your model from its config file or paper.

GPU VRAM requirement calculator

Workload

Efficiency

Both efficiency values are editable assumptions, not measurements. A benchmark of your own serving stack at your batch size is the best input.

API to compare

Formula

Cost per 1M tokens=GPU-hour price×Gtokens/s×3600×u×106\text{Cost per 1M tokens} = \dfrac{\text{GPU-hour price} \times G}{\text{tokens/s} \times 3600 \times u} \times 10^{6}
Show calculation steps (3)
  1. Decode step time (roofline)

    tstep=max⁡(Nb+BS‾kηGBW,B(2N+2nlayerS‾d)mGF)t_{\text{step}} = \max\left(\dfrac{N b + B \bar{S} k}{\eta\, G\, BW},\; \dfrac{B (2N + 2\, n_{\text{layer}} \bar{S} d)}{m\, G\, F}\right)

    One decode step reads the weights (N·b bytes) and the KV cache of every sequence (B·S̄·k bytes) at the achieved bandwidth η·G·BW, and does B·(2N + 2·n_layer·S̄·d) FLOPs at m·G·F. The slower of the two sets the step time (Williams, Waterman & Patterson\; Kaplan et al. 2020, eq. 2.2).

  2. KV cache per token

    k=2nlayernkvdnheadbkvk = 2\, {{n_{\text{layer}}}}\, {{n_{\text{kv}}}}\, \dfrac{d}{{{n_{\text{head}}}}}\, b_{\text{kv}}

    Keys and values for every layer: n_kv key/value heads of size d ÷ n_head, at b_kv bytes per value.

  3. Break-even monthly volume

    V∗=pGPU×G×730pAPI/106V^{*} = \dfrac{{{p_{\text{GPU}}}} \times G \times 730}{p_{\text{API}} / 10^{6}}

    Fixed monthly cost of an always-on deployment (price per GPU-hour × G × 730 h) divided by the API price per token at your prompt/output mix. Reachable only if it is below the capacity of one deployment.

Usage stats

Advertisement

Results

Self-hosted cost per 1M tokens
$1.75
input + output tokens, at your utilisation
API cost per 1M tokens
$2.00
Claude Haiku 4.5
Cheaper at your monthly volume: API
Monthly difference: $1,650
Break-even volume (always-on): 1,825 million tokens per month
Output tokens per second
397
estimate, whole deployment
Speed per user (decode)
31.5
tokens/s per sequence
All tokens per second
1,587
prompt + output
Self-hosted cost per 1M output tokens
$7.00
prompt processing included
API cost per 1M output tokens
$8.00
with the prompts they need
Capacity of one deployment
2,085
million tokens per month

Monthly cost at your volume

Monthly volume (million tokens) 1,000
GPU cost per hour (all GPUs) $5.00
Deployments needed 1
Self-hosted, always-on (730 h) $3,650.00
API $2,000.00

Cost curve

Monthly cost by monthly token volume 02,0004,0006,000 01,0002,000
Self-hosted (steps = more deployments) API Your volume Break-even

Horizontal: million tokens per month. Vertical: US dollars per month.

Throughput estimate

GPU NVIDIA H100 SXM5 80GB × 2
Decode is limited by memory bandwidth
Memory time per decode step (ms)
31.79
Compute time per decode step (ms)
2.88
Prefill time for one batch (ms) 4,275
Estimated output tokens per second 397
Peak dense TFLOPS used (per GPU) 989.4

Memory check

Model weights (GB) 140.0
KV cache per token (MB) 0.328
KV cache, all sequences at full length (GB) 10.5
Needed vs available (GB)
150.5 / 160
Fits in GPU memory yes
Most sequences that fit 30

Runtime buffers, activations and fragmentation also need memory, so keep some headroom.

API price used

Input price per 1M tokens $1.00
Output price per 1M tokens $5.00
Prompt tokens per output token 3

List price from BerriAI (LiteLLM), 2026-11-02

Other API models at the same prompt and output mix

Model Per 1M tokens Per month
Gemini 2.5 Flash-Lite $0.17 $175
Claude Haiku 5.5 $0.20 $200
gpt-6-luna $0.20 $200
DeepSeek Flash (off-peak) $0.26 $263
deepseek-flash $0.53 $525
Mistral Large $0.75 $750
Gemini 3.5 Flash-Lite $0.85 $850
Gemini 2.5 Flash $0.85 $850
DeepSeek V4 Pro (off-peak) $0.99 $990
Gemini 3.8 Flash $1.50 $1,500
deepseek-v4-pro $1.98 $1,980
Claude Haiku 4.5 $2.00 $2,000
gpt-6.1-terra $2.00 $2,000
Gemini 2.5 Pro $3.44 $3,438
Claude Sonnet 5.5 $4.00 $4,000
gpt-6.1-sol $4.00 $4,000
Gemini 4 Argon $4.00 $4,000
Gemini 3.1 Pro Preview $4.50 $4,500
Claude Sonnet 4.6 $6.00 $6,000
Claude Opus 5.5 $8.00 $8,000
Claude Opus 5 $10.00 $10,000
Claude Fable 5.1 $20.00 $20,000
gpt-6-astra $20.00 $20,000

Blue: cheaper than self-hosting at your volume. Orange: self-hosting is cheaper.

Estimates from datasheet peaks (dense, no sparsity) and your efficiency assumptions. Real throughput depends on the serving engine, kernels and traffic; enter a measured value for a decision.

Report an error on this page

What this calculator answers

Should you rent GPUs and run an open-weight model yourself, or pay an API per token? The answer rests on three numbers: how many tokens per second your GPUs really produce, how busy you keep them, and how many tokens you need each month. This calculator estimates the first from the GPU datasheet and the shape of the model, turns it into a cost per 1M tokens, and compares it with an API price for the same mix of prompt and output tokens.

How the throughput is estimated

Generating one token for every sequence in a batch (a decode step) reads all model weights from GPU memory once, plus the key/value (KV) cache of each sequence. It also costs about \( 2N \) floating-point operations per token for a model with \( N \) parameters, plus an attention term that grows with the context (Kaplan et al., 2020, eq. 2.2). The roofline model of Williams, Waterman and Patterson says a step takes as long as the slower of the two limits:

$$ t_{\text{step}} = \max\left( \frac{N b + B \bar{S} k}{\eta \, G \, BW},\; \frac{B \, (2N + 2 \, n_{\text{layer}} \bar{S} d)}{m \, G \, F} \right) $$

\( b \) is bytes per weight, \( B \) the number of concurrent sequences, \( \bar{S} \) the average context while decoding (prompt plus half the output), \( k \) the KV cache bytes per token, \( G \) the number of GPUs, \( BW \) the memory bandwidth, \( F \) the dense peak FLOPS for the precision, \( \eta \) the bandwidth efficiency and \( m \) the compute utilisation (MFU). The KV cache per token is \( k = 2 \, n_{\text{layer}} \, n_{\text{kv}} \, (d / n_{\text{head}}) \, b_{\text{kv}} \).

The prompt is processed first and in parallel (prefill), so it is usually limited by compute. As in Pope et al. (2022), one batch takes the prefill time plus one decode step per output token, and output tokens per second equal \( B \) times the output tokens per request divided by that batch time.

At small batches the memory term dominates and the speed per user is close to bandwidth divided by model size. Larger batches share each read of the weights across more sequences, so total throughput rises until the KV cache or compute becomes the limit.

GPU illustration with memory split between model weights and KV cache, and a bandwidth pipe feeding the compute chip
Decode speed mostly depends on how fast the GPU can stream model weights from memory, so memory bandwidth sets the throughput ceiling.

From tokens per second to cost per 1M tokens

$$ \text{Cost per 1M} = \frac{p_{\text{GPU}} \times G}{\text{tokens/s} \times 3600 \times u} \times 10^{6} $$

\( p_{\text{GPU}} \) is the price per GPU-hour and \( u \) the average utilisation: the share of paid hours in which the GPUs serve at the modelled rate. A deployment that is busy half the time pays twice as much per token. The API side weights its input and output prices by your prompt and output tokens per request, so both sides are priced on the same tokens.

Advertisement
Flow diagram showing GPU-hour price and GPU count, scaled by utilization and throughput, turning into cost per million tokens
Self-hosted cost per 1M tokens: the GPU-hour cost divided by the tokens actually produced each hour at your utilization.

Break-even volume

An always-on deployment costs the same every month: price per GPU-hour × GPUs × 730 hours. The API bill grows with volume. Break-even is the monthly volume at which the two are equal, the fixed monthly cost divided by the API price per token. It only counts if one deployment can serve that volume beyond its capacity you need another deployment, which is why the self-hosted line in the chart rises in steps.

With the default inputs (Llama 3 70B in BF16 on two H100 SXM GPUs at an example 2.50 USD per GPU-hour, 16 concurrent sequences, 1,500 prompt and 500 output tokens per request) the estimate is about 397 output tokens per second and 1.75 USD per 1M tokens at 50% utilisation. Against an example API price of 1 USD input and 5 USD output per 1M tokens, break-even is about 1,825 million tokens a month.

Advertisement
Line chart where a rising API cost line crosses a flat self-hosted cost line at the break-even volume
Break-even: once monthly volume passes the crossing point, a fixed-cost GPU becomes cheaper than paying the API per token.

GPU figures used

The presets use each vendor's own datasheet. Tensor throughput is often quoted with structured sparsity, which is twice the dense rate ordinary inference runs dense, so the table and the calculator use the dense figures.

GPU Memory (GB) Bandwidth (GB/s) Dense FP16/BF16 TFLOPS Dense FP8 TFLOPS
NVIDIA H100 SXM5 80 3,352 989.4 1,978.9
NVIDIA H100 PCIe 80 2,039 756 1,513
NVIDIA H200 SXM 141 4,800 989.5 1,979
NVIDIA A100 80GB SXM 80 2,039 312 —
NVIDIA A100 80GB PCIe 80 1,935 312 —
NVIDIA L40S 48 864 362.05 733
NVIDIA L4 24 300 121 242.5
AMD Instinct MI300X 192 5,300 1,307.45 2,614.9

For a GPU that is not listed, choose Custom GPU and enter its memory, bandwidth and dense TFLOPS.

What the estimate leaves out

  • Serving engines differ: continuous batching, paged KV cache, speculative decoding and tuned kernels can change throughput a lot. Measure your own stack and enter the result.
  • Splitting a model across GPUs adds communication time that the estimate ignores.
  • A mixture-of-experts model reads only its active experts per token: enter the active parameters for speed, but check memory with the total.
  • Models use different tokenizers, so the same text can be a different number of tokens.
  • Power, staff time, spare capacity for peaks and network egress are not included unless you add them to the hourly price.

To size GPU memory in more detail, use the GPU VRAM requirement calculator.

Sources & References

Last updated:

Most popular in Industry-Specific

View all Industry-Specific calculators