What this calculator answers
Should you rent GPUs and run an open-weight model yourself, or pay an API per token? The answer rests on three numbers: how many tokens per second your GPUs really produce, how busy you keep them, and how many tokens you need each month. This calculator estimates the first from the GPU datasheet and the shape of the model, turns it into a cost per 1M tokens, and compares it with an API price for the same mix of prompt and output tokens.
How the throughput is estimated
Generating one token for every sequence in a batch (a decode step) reads all model weights from GPU memory once, plus the key/value (KV) cache of each sequence. It also costs about \( 2N \) floating-point operations per token for a model with \( N \) parameters, plus an attention term that grows with the context (Kaplan et al., 2020, eq. 2.2). The roofline model of Williams, Waterman and Patterson says a step takes as long as the slower of the two limits:
$$ t_{\text{step}} = \max\left( \frac{N b + B \bar{S} k}{\eta \, G \, BW},\; \frac{B \, (2N + 2 \, n_{\text{layer}} \bar{S} d)}{m \, G \, F} \right) $$\( b \) is bytes per weight, \( B \) the number of concurrent sequences, \( \bar{S} \) the average context while decoding (prompt plus half the output), \( k \) the KV cache bytes per token, \( G \) the number of GPUs, \( BW \) the memory bandwidth, \( F \) the dense peak FLOPS for the precision, \( \eta \) the bandwidth efficiency and \( m \) the compute utilisation (MFU). The KV cache per token is \( k = 2 \, n_{\text{layer}} \, n_{\text{kv}} \, (d / n_{\text{head}}) \, b_{\text{kv}} \).
The prompt is processed first and in parallel (prefill), so it is usually limited by compute. As in Pope et al. (2022), one batch takes the prefill time plus one decode step per output token, and output tokens per second equal \( B \) times the output tokens per request divided by that batch time.
At small batches the memory term dominates and the speed per user is close to bandwidth divided by model size. Larger batches share each read of the weights across more sequences, so total throughput rises until the KV cache or compute becomes the limit.
From tokens per second to cost per 1M tokens
$$ \text{Cost per 1M} = \frac{p_{\text{GPU}} \times G}{\text{tokens/s} \times 3600 \times u} \times 10^{6} $$\( p_{\text{GPU}} \) is the price per GPU-hour and \( u \) the average utilisation: the share of paid hours in which the GPUs serve at the modelled rate. A deployment that is busy half the time pays twice as much per token. The API side weights its input and output prices by your prompt and output tokens per request, so both sides are priced on the same tokens.
Break-even volume
An always-on deployment costs the same every month: price per GPU-hour × GPUs × 730 hours. The API bill grows with volume. Break-even is the monthly volume at which the two are equal, the fixed monthly cost divided by the API price per token. It only counts if one deployment can serve that volume beyond its capacity you need another deployment, which is why the self-hosted line in the chart rises in steps.
With the default inputs (Llama 3 70B in BF16 on two H100 SXM GPUs at an example 2.50 USD per GPU-hour, 16 concurrent sequences, 1,500 prompt and 500 output tokens per request) the estimate is about 397 output tokens per second and 1.75 USD per 1M tokens at 50% utilisation. Against an example API price of 1 USD input and 5 USD output per 1M tokens, break-even is about 1,825 million tokens a month.
GPU figures used
The presets use each vendor's own datasheet. Tensor throughput is often quoted with structured sparsity, which is twice the dense rate ordinary inference runs dense, so the table and the calculator use the dense figures.
| GPU | Memory (GB) | Bandwidth (GB/s) | Dense FP16/BF16 TFLOPS | Dense FP8 TFLOPS |
|---|---|---|---|---|
| NVIDIA H100 SXM5 | 80 | 3,352 | 989.4 | 1,978.9 |
| NVIDIA H100 PCIe | 80 | 2,039 | 756 | 1,513 |
| NVIDIA H200 SXM | 141 | 4,800 | 989.5 | 1,979 |
| NVIDIA A100 80GB SXM | 80 | 2,039 | 312 | — |
| NVIDIA A100 80GB PCIe | 80 | 1,935 | 312 | — |
| NVIDIA L40S | 48 | 864 | 362.05 | 733 |
| NVIDIA L4 | 24 | 300 | 121 | 242.5 |
| AMD Instinct MI300X | 192 | 5,300 | 1,307.45 | 2,614.9 |
For a GPU that is not listed, choose Custom GPU and enter its memory, bandwidth and dense TFLOPS.
What the estimate leaves out
- Serving engines differ: continuous batching, paged KV cache, speculative decoding and tuned kernels can change throughput a lot. Measure your own stack and enter the result.
- Splitting a model across GPUs adds communication time that the estimate ignores.
- A mixture-of-experts model reads only its active experts per token: enter the active parameters for speed, but check memory with the total.
- Models use different tokenizers, so the same text can be a different number of tokens.
- Power, staff time, spare capacity for peaks and network egress are not included unless you add them to the hourly price.
To size GPU memory in more detail, use the GPU VRAM requirement calculator.