VRAM math for any open LLM, from 7B to 405B

The question we get on day one of every engagement: will this model fit on my GPU? The answer is a closed-form calculation, not a lookup table. Once you've memorized the formula you can size any model against any hardware in 30 seconds.

The formula has three components: weights, KV cache, and overhead. We'll work through each, then run the calculation on real examples.

Component 1: Weights

The dominant memory cost. A model with P parameters at B bits per parameter takes P × B / 8 bytes.

For common cases:

Quant	Bits/param	Bytes for a 7B	Bytes for a 70B
bf16 / fp16	16	14 GB	140 GB
Q8_0	~8.5	7.4 GB	74 GB
Q6_K	~6.6	5.8 GB	58 GB
Q5_K_M	~5.7	5.0 GB	50 GB
Q4_K_M	~4.85	4.3 GB	43 GB
Q3_K_S	~3.4	3.0 GB	30 GB
Q2_K	~2.9	2.5 GB	25 GB

The asymmetric quants (Q4_K_M, Q5_K_M) aren't exact bit counts because they allocate more bits to some weight groups (attention) and fewer to others (mlp). The averages above are reliable to ~5%.

Component 2: KV cache

For each token in your context window, the model stores a key and a value vector per attention head per layer. The formula:

kv_bytes = 2 × context_length × n_layers × n_kv_heads × head_dim × bytes_per_param

Where bytes_per_param is 2 for fp16/bf16 KV (the default) or 1 for Q8_0 KV (with --cache-type-k q8_0) or 0.5 for Q4_0 KV.

For modern models with grouped-query attention (GQA), n_kv_heads is much smaller than n_attention_heads. This drastically reduces KV cache size. Llama-3.3 70B has 64 attention heads but only 8 KV heads — an 8× cache reduction over a model without GQA.

Llama-3.3 70B has 80 layers, 8 KV heads, 128 head dim. At 8K context, bf16 KV cache:

2 × 8192 × 80 × 8 × 128 × 2 = 2.7 GB

Quantizing KV cache to Q8_0 halves this. To Q4_0 quarters it. The accuracy hit from Q8_0 KV is negligible; Q4_0 KV is noticeable but tolerable for many workloads.

Component 3: Overhead

The third bucket: compute buffers, activations during the forward pass, alignment padding, and the inference engine's bookkeeping. Empirically:

llama.cpp: ~800 MB - 1.5 GB overhead, mostly constant
vLLM: ~2-3 GB overhead, more variable based on batch size

We use 1.5 GB as a round number for llama.cpp and 2.5 GB for vLLM in initial sizing. Real measurements with nvidia-smi always come in within 10%.

The formula

total_vram = weights + kv_cache + overhead

With safety margin:

required_vram = total_vram × 1.10

The 10% buffer absorbs runtime fluctuation (kernel launches, allocator fragmentation, occasional spikes). Don't size to the millimeter — production OOMs at 3 AM are not worth saving 1 GB.

Worked examples

Llama-3.3 8B Instruct at Q4_K_M, 8K context, llama.cpp:

Weights: 8B × 4.85 / 8 = 4.85 GB
KV: 2 × 8192 × 32 × 8 × 128 × 2 / 1e9 = 1.07 GB
Overhead: 1.5 GB
Total: 7.4 GB
With 10%: 8.1 GB → fits comfortably on a 12 GB card, with room for a small prompt cache.

Qwen 2.5 32B Instruct at Q5_K_M, 16K context, llama.cpp:

Weights: 32B × 5.7 / 8 = 22.8 GB
KV: 2 × 16384 × 64 × 8 × 128 × 2 / 1e9 = 4.3 GB
Overhead: 1.5 GB
Total: 28.6 GB
With 10%: 31.5 GB → fits on a 40 GB A100 or 48 GB A6000, NOT on a 24 GB 4090.

Llama-3.3 70B Instruct at Q4_K_M, 8K context, llama.cpp:

Weights: 70B × 4.85 / 8 = 42.4 GB
KV: 2 × 8192 × 80 × 8 × 128 × 2 / 1e9 = 2.7 GB
Overhead: 1.5 GB
Total: 46.6 GB
With 10%: 51.3 GB → needs a 80 GB H100 (with massive headroom), or 2× A100 40GB with tensor parallelism, or 1× A6000 48GB very tight.

Llama-3.3 70B Instruct at Q3_K_S, 4K context, llama.cpp:

Weights: 70B × 3.4 / 8 = 29.75 GB
KV: 2 × 4096 × 80 × 8 × 128 × 2 / 1e9 = 1.34 GB
Overhead: 1.5 GB
Total: 32.6 GB
With 10%: 35.9 GB → fits on a 40 GB A100 with a few GB to spare.

DeepSeek-V3 (671B MoE, 37B active) at Q4_K_M, 8K context:

Weights: 671B × 4.85 / 8 = 407 GB (MoE — all experts loaded)
KV: similar to a 37B dense at the same context
Overhead: 2-3 GB
Needs multi-GPU. A single 80 GB H100 isn't close. Tensor or expert parallelism across 4-8 H100s is the typical setup.

Doesn't fit? Knobs in order of impact

Lower quant. Q4_K_M → Q3_K_S can save 30%+ on weight memory. Q3 to Q2 a similar amount. Quality drops are noticeable below Q3; we don't ship Q2 without imatrix calibration.
Shorter context. KV scales linearly with context length. Cutting from 32K to 8K saves a lot on big models. If your prompts genuinely don't need 32K, don't pay for it.
Quantize KV cache. --cache-type-k q8_0 --cache-type-v q8_0 halves KV memory. Try this before going below Q4_K_M on weights.
Smaller model. Sometimes the right answer is dropping from 70B to 32B and recovering the lost quality with LoRA on your domain.
Tensor parallelism. Split weights across 2 or 4 GPUs. Adds inter-GPU bandwidth pressure but unlocks bigger models.

Why not just install and see?

Sizing matters before you spend $4/hour on an H100 rental. It also matters for hardware purchasing decisions (do you buy 2× A100 40GB or 1× A100 80GB?). And it tells you the safety margin you have for batch size and concurrent users — a 70B that "just barely fits" at concurrency 1 won't survive concurrency 8.

The formula is reliable to within 5-10%. Real measurements will match it. Once you've internalized the components, sizing decisions become an arithmetic exercise instead of a guessing game.