When llama.cpp beats vLLM (and vice versa)

llama.cpp and vLLM are both excellent inference engines for open LLMs. They are not interchangeable. We've shipped engagements where switching engines doubled throughput, and engagements where the same switch halved it. The right answer depends on your hardware, your traffic shape, and your latency tolerance.

This post is the decision tree we apply when picking an engine for a new deployment.

The architectural difference in one paragraph

llama.cpp is a C++ inference runtime optimized for single-request, low-latency workloads on heterogeneous hardware including CPU, Apple Silicon, AMD ROCm, and Nvidia CUDA. It supports aggressive quantization (down to 1.5-bit), runs efficiently on consumer GPUs, and scales down to laptops and Raspberry Pis.

vLLM is a Python+CUDA inference runtime optimized for high-throughput, batched workloads on Nvidia GPUs. Its PagedAttention implementation packs many concurrent requests into the same GPU memory efficiently. It assumes data-center hardware and isn't designed for laptops.

If your decision were just "single user vs many users," the answer would be obvious. It rarely is.

When llama.cpp wins

Concurrent users under ~4. llama.cpp's per-request latency at low concurrency beats vLLM's because vLLM is built around continuous batching — there's overhead to the batch machinery that doesn't pay off until you have many concurrent requests. For an internal tool with 1-2 users, llama.cpp is faster.

Consumer GPUs (4090, 7900 XT, M-series Macs). vLLM works best on H100-class hardware. On a 4090, llama.cpp's Q4_K_M throughput beats vLLM's bfloat16 throughput because the quantization fits in less VRAM and the kernels are well-tuned for Ada Lovelace. We've measured 1.5-2× advantage to llama.cpp on a 4090 serving a 7B model at concurrency 1-2.

Aggressive quantization required. vLLM supports 4-bit (AWQ, GPTQ) but llama.cpp goes further — Q3_K_S, Q2_K, IQ-quants. If you're trying to fit a 70B on a single 24GB card, llama.cpp's IQ2_XS is one of the few options. Quality is rough at that level but it runs.

Cross-platform requirement. Need the same model to run on a developer's M2 MacBook and a Linux server? llama.cpp produces the same GGUF that runs on both. vLLM runs only on Linux + CUDA.

LoRA adapter swapping at runtime. llama.cpp's --lora-scaled <path> <scale> lets you load and unload LoRA adapters per request without restarting. vLLM supports this too but the implementation is heavier.

When vLLM wins

Many concurrent users. This is the headline. vLLM's continuous batching keeps the GPU at high utilization across dozens of concurrent requests. At concurrency 16+, vLLM's tokens-per-second on the same H100 will be 5-10× llama.cpp's.

Sustained traffic. llama.cpp's request handling has more per-request overhead. Under sustained load, that overhead compounds. vLLM amortizes it across the batch.

Strict throughput SLAs. If your application needs guaranteed tokens-per-second under load, vLLM's batching is the only way to get there without massive over-provisioning.

You have data-center GPUs. A100, H100, B100 — vLLM's kernels are tuned for these. Less true on consumer cards. On a 4090, the advantage shrinks; on an H100, vLLM is in its element.

Speculative decoding. vLLM has robust support for speculative decoding with a smaller draft model. llama.cpp has it but the integration is more brittle.

Specific scenarios

Single-tenant chat for a small internal team (5-20 users, intermittent traffic). llama.cpp on a 4090 with Q4_K_M. Cheap, fast at low concurrency, easy to operate. We've shipped a half-dozen of these.

Customer-facing chat with 100+ concurrent users. vLLM on an H100 or A100 with bfloat16 weights, continuous batching, prompt prefix caching enabled. Real-time latency, sustained throughput.

Batch inference (classifying 10M documents overnight). vLLM. The batching engine is exactly what you want here. llama.cpp can do it but you'll under-utilize the GPU.

Edge deployment (laptops, on-prem boxes with mixed hardware). llama.cpp. Cross-platform support, low memory footprint, runs on whatever's there. We had one engagement where the production target was a customer's Windows server with no GPU; llama.cpp on CPU at Q5_K_M was the only viable answer.

LoRA-per-tenant routing. llama.cpp, with adapters hot-swapped per request. vLLM's LoRA multiplexing is improving but llama.cpp's --lora-scaled has been stable for two years.

Performance numbers

Indicative measurements on a 7B Llama-3-Instruct, serving concurrent chat requests. Hardware: 1× A100 40GB.

At concurrency 1, llama.cpp is 36% faster. At concurrency 16, vLLM is 4.4× faster. The crossover is somewhere around concurrency 3-6 depending on prompt length and model.

What about TensorRT-LLM?

TensorRT-LLM is the throughput king on Nvidia hardware when you're willing to invest in engine compilation. We've seen it beat vLLM by 20-40% on H100 for the same workload. The catch:

• Compilation overhead. Each model + each quantization + each context length needs its own pre-compiled engine plan. A single recipe that "just works" for any model doesn't exist; you have to build per-deployment.
• Tight Nvidia coupling. TensorRT-LLM is even more Nvidia-specific than vLLM.
• Operational complexity. Restarting with different parameters means a new engine compile.

We pick TensorRT-LLM when:

• The deployment is a single large workload that justifies the engineering investment
• The GPU is H100 / B100 (the kernels target the latest architectures)
• The team has capacity to maintain pre-built engines

For most engagements, vLLM is the right ceiling and TensorRT-LLM is the next step if you're hitting that ceiling.

Decision tree

Need cross-platform / consumer GPU / CPU fallback?
  → llama.cpp

Concurrency ≥ ~6 sustained?
  → vLLM (H100 / A100 / RTX 6000)
    → If throughput is critical and you can invest in engine builds:
      → TensorRT-LLM

Otherwise (concurrency 1-5, single-tenant, fast iteration):
  → llama.cpp on consumer GPU

Migration

Engines aren't a one-way decision. The same fine-tuned model can serve through llama.cpp (GGUF) and vLLM (safetensors) — adapters convert. The main cost of switching is operational: different config files, different ports, different metrics formats, different LoRA semantics.

This is where the multi-engine abstraction matters. If your deployment hardware changes (you outgrow a 4090 and rent an H100), you want to swap engines without changing your application. A control plane that speaks both engines' configuration languages saves a week of cross-cutting changes.