VRAM math

Why "VRAM" is more than weights

A common shortcut: "a 7B Q4 model is 4 GB so any 8 GB GPU works." That's the weights size — the bytes the GGUF takes on disk. Actual VRAM during inference adds three more components on top:

1. KV cache — per-layer key + value tensors for every token in the active context. Grows linearly with context length and proportional to num_layers * hidden_size. For a 7B at 4k context, this is roughly 1-2 GB. At 32k context, it's 8-16 GB and starts dominating the weights.
2. Activations — intermediate tensors during the forward pass. Roughly batch_size * hidden_size * sequence_length. Small for single-stream inference; large for batched serving.
3. CUDA overhead — kernels, cuDNN workspace, fragmentation. 1-2 GB even for a quiet model.

The VRAM breakdown bar on the Hardware tab labels each component so you can see where the budget is going. The total is what you need free on the GPU at runtime, not what the model "is."

The math

For dense transformer models:

weights_bytes      = params * bytes_per_weight
                     # Q4 ≈ 0.55 bytes/weight, Q8 ≈ 1 byte, FP16 ≈ 2 bytes
kv_cache_bytes     = 2 * num_layers * (hidden_size / num_attn_heads) * num_kv_heads
                     * context_length * batch_size * dtype_bytes
                     # dtype_bytes is FP16 (2) unless KV cache quantization is on
activations_bytes  = batch_size * context_length * hidden_size * dtype_bytes * factor
                     # factor ~4-8 depending on the kernel choice
overhead_bytes     = 1.5 GB

Blueprint pulls params, num_layers, num_attn_heads, num_kv_heads, and hidden_size from the kernel catalog (which mirrors the model's config.json on HuggingFace), then plugs in your chosen quant + context

• batch. The number is real; it's not an estimate from a regression on benchmarks.

What-if sliders

The Hardware tab's What-if sliders let you change quant / context / batch and watch the VRAM number move. Use them to find the largest context your GPU can serve, or to decide whether dropping from Q4 to Q3 buys you the context length you actually need.

Common pitfalls

• Forgetting batch > 1. If you're serving multiple concurrent requests, the KV cache scales linearly. A 7B Q4 that serves 1 request at 4k context in 5 GB needs 20 GB to serve 4 of those at once.
• Trusting "model size" from HF. The Hugging Face card lists weight size; that's one of four components. Always run it through the VRAM bar.
• Ignoring fragmentation on long-running servers. After hours of uptime, CUDA's allocator wastes ~5-10% to fragmentation. Add headroom if you're targeting 24/7 serving.