Choosing an engine

The three engines

Blueprint's engine selector picks which backend serves your model:

• llama.cpp (default, always available) — C++ binary, single-file install, supports CPU + every GPU vendor, runs GGUF weights. Throughput is competitive but not best-in-class.
• vLLM (opt-in via Python runtime) — Python-based engine with PagedAttention. ~1.5-2× llama.cpp's throughput on the same GPU for batched workloads. Requires the Python sidecar + an NVIDIA GPU.
• TensorRT-LLM (opt-in via Python runtime) — NVIDIA's ahead-of-time compiled engine. Highest absolute throughput on H100/H200 with a fixed batch profile. Engine plan is GPU-model- specific (the .engine file you build on an H100 doesn't run on an A100).

The picker on the Service card shows which engines are installed and recommends one based on your serve config.

When to use each

llama.cpp — default for almost everyone:

• Laptop, workstation, or small server
• Single-user or low-concurrency serving
• GGUF model + you want one binary doing everything
• Any non-NVIDIA hardware (AMD, Apple Silicon, CPU-only)

vLLM — switch when:

• You're on an A100 / H100 / L40S and serving batched traffic (multiple concurrent requests sharing the GPU)
• Throughput is your bottleneck and llama.cpp's metrics show GPU underutilized (KV cache not full)
• You have the disk + time for the Python sidecar + a 2.5 GB PyTorch install

TensorRT-LLM — switch when:

• You're on an H100 / H200 with a known fixed batch profile
• You've already squeezed everything out of vLLM and need the last 10-20% throughput
• You're OK rebuilding the engine plan whenever you upgrade GPU generations

What changes when you switch

Switching from llama.cpp → vLLM doesn't just change the binary, it changes some semantics:

• Model format — vLLM expects a HuggingFace identifier (meta-llama/Llama-3.2-3B-Instruct), not a GGUF path. It downloads the safetensors weights through HF, NOT your existing GGUF.
• Tokenizer — vLLM uses HF's tokenizer; outputs can differ very slightly from llama.cpp on the same prompt.
• Sampling — temperature, top_p, etc. are passed through but the underlying implementation differs; identical params can produce different outputs.

If your prompts have been tuned against llama.cpp's behavior, expect to re-tune after switching.

Installing the Python-side engines

1. Dashboard → Python runtime card.
2. Install Python core + PyTorch + CUDA if not already there.
3. Tick vLLM engine or TensorRT-LLM engine and Install.
4. The engine appears in the Service card's engine picker as "Installed."

For TensorRT-LLM specifically, you still need to run trtllm-build to produce the engine plan before serving. The Service card's TensorRT-LLM option points you at the right command.

Common pitfalls

• Switching engines without re-evaluating: a model that scored well on your eval set under llama.cpp may score differently under vLLM (tokenizer differences). Re-run the eval after switching.
• vLLM on consumer GPUs: vLLM is optimized for datacenter cards with high memory bandwidth. On an RTX 4090 it can actually be slightly slower than llama.cpp for single-user workloads. Bench before committing.
• TensorRT-LLM engine plan staleness: rebuild the plan whenever you change model, batch profile, or GPU generation. Old plans silently load and produce wrong outputs.