Does tensor parallelism cut memory exactly in half on two GPUs?

Almost. Each GPU holds half of every layer's weights and half the KV cache, so memory per card is roughly total divided by N. There is a small fixed overhead — communication buffers, and a few tensors like layer norms and embeddings that are replicated rather than split — worth around 5 to 10 percent per card and it does not shrink as you add GPUs.

Do the GPUs need NVLink?

For tensor parallelism, a fast link matters a lot because every layer exchanges activations between cards on every token. NVLink is ideal. PCIe 4.0 x16 works but costs you throughput. PCIe x4 or x1 risers will bottleneck badly. Pipeline parallelism is far less link-sensitive but has its own trade-offs.

What is the difference between tensor, pipeline and ZeRO parallelism?

Tensor parallelism splits each layer across GPUs — low latency, needs a fast link, usually within one node. Pipeline parallelism puts whole layers on different GPUs and streams micro-batches through — link-tolerant, adds latency. ZeRO and FSDP shard the optimizer, gradients and optionally weights for training, and gather them just in time. Real large-scale setups combine them.

Can I mix different GPU models?

Technically sometimes, practically avoid it. Tensor parallelism assumes equal cards; a slower or smaller GPU becomes the bottleneck for the whole group. Use identical cards.

// multi-gpu

Multi-GPU LLM Split Calculator

How many GPUs you need to hold a model, and the VRAM per card at tensor-parallel 2, 4 and 8.

Model

GPU

QuantizationContexttokens

Total VRAM (all GPUs together)

762.4GB

Min. H100 SXM 80GB to fit

11GPUs

Tensor-parallel	VRAM per GPU	Fits 80GB?
TP = 1	815.8 GB	no
TP = 2	407.9 GB	no
TP = 4	203.9 GB	no
TP = 8	102.0 GB	no

per-GPU ≈ total ÷ N × (1 + 0.07) — comms buffers add ~7% that doesn't shrink with N

Presets verified 2026-09. Tensor parallelism splits every layer's weights and KV cache evenly across N GPUs on one node; it needs a fast link (NVLink, or PCIe 4.0 x16). The ~7% overhead is a rough allowance for communication buffers and duplicated norms/embeddings — it does not fall as N rises. Pipeline parallelism and ZeRO/FSDP have different trade-offs.

notes

What this computes

You have a model that will not fit on one card. This tool answers two questions:

How many of GPU X do I need? It walks N upward until total VRAM ÷ N × (1 + overhead) fits in one card’s memory.
What does the per-GPU load look like at TP = 1, 2, 4, 8? A table so you can see where it crosses your card’s capacity.

The total VRAM is the same weights + KV cache + overhead sum used elsewhere in this suite, at your chosen quantization and context length.

Why the overhead doesn’t disappear

Tensor parallelism splits the big weight matrices and the KV cache evenly. But each GPU also keeps a full copy of the small things — token embeddings, the final norm, RoPE tables — and a set of NCCL communication buffers. That fixed cost is a few percent of the model and it is per GPU, so going from 2 to 4 cards halves the split part but not the overhead part.

Worked example

Llama 3.1 405B at fp16, 8k context ≈ 810 GB total.

On H100 80GB (usable ~75 GB after the 6% haircut): 810 ÷ N × 1.07 ≤ 75 → N = 12. In practice you round to 16 (2 nodes of 8) for a clean tensor-parallel size and headroom.
At Q4_K_M the same model is ~250 GB → N = 4 H100s, one node.

Notes

Presets carry a verified date. This models tensor parallelism on one node with an even split; pipeline parallelism, expert parallelism for MoE, and ZeRO/FSDP sharding for training follow different arithmetic.

faq

Does tensor parallelism cut memory exactly in half on two GPUs?: Almost. Each GPU holds half of every layer's weights and half the KV cache, so memory per card is roughly total divided by N. There is a small fixed overhead — communication buffers, and a few tensors like layer norms and embeddings that are replicated rather than split — worth around 5 to 10 percent per card and it does not shrink as you add GPUs.
Do the GPUs need NVLink?: For tensor parallelism, a fast link matters a lot because every layer exchanges activations between cards on every token. NVLink is ideal. PCIe 4.0 x16 works but costs you throughput. PCIe x4 or x1 risers will bottleneck badly. Pipeline parallelism is far less link-sensitive but has its own trade-offs.
What is the difference between tensor, pipeline and ZeRO parallelism?: Tensor parallelism splits each layer across GPUs — low latency, needs a fast link, usually within one node. Pipeline parallelism puts whole layers on different GPUs and streams micro-batches through — link-tolerant, adds latency. ZeRO and FSDP shard the optimizer, gradients and optionally weights for training, and gather them just in time. Real large-scale setups combine them.
Can I mix different GPU models?: Technically sometimes, practically avoid it. Tensor parallelism assumes equal cards; a slower or smaller GPU becomes the bottleneck for the whole group. Use identical cards.