How does a server like vLLM decide how many requests it can run at once?

It loads the weights once, reserves a little for activations and framework, and turns the rest of VRAM into a KV-cache pool. Each in-flight sequence draws from that pool in proportion to its context length. When the pool is full, new requests wait in a queue. So concurrency is set by free VRAM divided by KV cache per request.

Why does aggregate throughput go up with batching but per-request speed does not?

Generating one token requires reading the model weights from memory. If eight requests generate their next token together, that single weight read serves all eight — the expensive part is amortised. Each individual request still advances at roughly the single-stream rate, but the server's total tokens per second climbs, sub-linearly, until it hits a compute or memory-bandwidth ceiling.

Does every request really use its full context?

Rarely. Most requests are far shorter than the max context. Servers use paged KV cache so a 500-token request only occupies 500 tokens' worth of pool, not the maximum. That means real concurrency is usually well above the worst-case number this tool shows for a fixed context.

Can I fit more requests by quantizing the KV cache?

Yes. An fp8 or Q8 KV cache halves the per-request cost, roughly doubling concurrency, for a small quality impact. vLLM and TensorRT-LLM both support it.

// serving

LLM Serving Concurrency Calculator

How many concurrent requests a GPU can serve — from the VRAM left after the weights, divided by the KV cache per request.

GPU

GPU count

Model

Quantization

Context per requesttokKV precision

Max concurrent requests

113seqs

KV cache per request

0.5GB

Aggregate throughput

4,665tok/s

free after weights = 56.6 GB ÷ 0.5 GB/seq = 113 concurrent sequences

Single-stream decode: 95 tok/s. Batching many requests amortizes the weight reads, so aggregate throughput climbs above that — at the cost of KV-cache memory and per-request latency.

Presets verified 2026-09. Models a vLLM-style server: weights load once, remaining VRAM becomes a KV-cache pool shared by all in-flight requests. The concurrency number is memory-limited; real throughput also depends on the scheduler, chunked prefill, and whether requests actually use their full context. Aggregate throughput here is a rough sub-linear estimate, not a benchmark.

notes

The calculation

free VRAM        = total VRAM − weights − framework/activation reserve
KV per request   = context × 2 × layers × kv_heads × head_dim × bytes
max concurrent   = free VRAM ÷ KV per request

The weights load once regardless of load. Everything after that is the KV-cache pool, and it is shared by every request currently generating.

Reading the result

Max concurrent requests — how many sequences of the chosen context length fit in the pool at once. Treat it as a worst case: real traffic has mixed, mostly-shorter contexts, so a paged server handles more.
KV cache per request — the pool cost of one sequence. If this is large, either shorten the served context, quantize the KV cache, or pick a model with stronger grouped-query attention.
Aggregate throughput — a rough sub-linear estimate of the server’s total tokens per second once batching kicks in. Not a benchmark.

Worked example

Llama 3.1 8B at fp16 on one A100 80GB, 4k served context, fp16 KV:

Weights ≈ 16 GB. Reserve ≈ 2 GB. Free ≈ 55 GB (after vLLM’s ~8% headroom).
KV per request: 4096 × 2 × 32 × 8 × 128 × 2 ≈ 0.5 GB.
Max concurrent ≈ 55 ÷ 0.5 ≈ 110 requests.
Halve it with an fp8 KV cache? Roughly 220.

Notes

Presets carry a verified date. This models a paged-attention server; throughput also depends on the scheduler, chunked prefill, speculative decoding and how bursty your traffic is.

faq

How does a server like vLLM decide how many requests it can run at once?: It loads the weights once, reserves a little for activations and framework, and turns the rest of VRAM into a KV-cache pool. Each in-flight sequence draws from that pool in proportion to its context length. When the pool is full, new requests wait in a queue. So concurrency is set by free VRAM divided by KV cache per request.
Why does aggregate throughput go up with batching but per-request speed does not?: Generating one token requires reading the model weights from memory. If eight requests generate their next token together, that single weight read serves all eight — the expensive part is amortised. Each individual request still advances at roughly the single-stream rate, but the server's total tokens per second climbs, sub-linearly, until it hits a compute or memory-bandwidth ceiling.
Does every request really use its full context?: Rarely. Most requests are far shorter than the max context. Servers use paged KV cache so a 500-token request only occupies 500 tokens' worth of pool, not the maximum. That means real concurrency is usually well above the worst-case number this tool shows for a fixed context.
Can I fit more requests by quantizing the KV cache?: Yes. An fp8 or Q8 KV cache halves the per-request cost, roughly doubling concurrency, for a small quality impact. vLLM and TensorRT-LLM both support it.