← All articles · Deploy
Installing the runtime
The runtime is llama.cpp — Blueprint downloads the pre-built binary for your OS and verifies it before letting you serve anything.
2 min read
What "runtime" means here
Blueprint's default inference engine is llama.cpp's llama-server
— the same binary you'd build yourself by cloning the
ggerganov/llama.cpp repo and running make. It's a single C++
executable that loads GGUF model files and serves an OpenAI-compatible
HTTP API on a port you choose.
Blueprint downloads a pre-built release from llama.cpp's GitHub releases page — no compilation on your machine, no CUDA toolkit required. The right build is picked for your OS + GPU vendor:
- Windows + NVIDIA: CUDA-enabled build
- Linux + NVIDIA: CUDA-enabled build
- Linux + AMD: ROCm build
- macOS Apple Silicon: Metal-enabled build
- macOS Intel: CPU-only build
The runtime lives at ~/.blueprint/runtime/llama-server[.exe]. Updates
happen on demand from the Maintain tab.
Why we ship llama.cpp specifically
Among the open-engine choices (llama.cpp, vLLM, TensorRT-LLM, exllama, MLC, etc.), llama.cpp is the broadest-compatibility option:
- Runs anywhere: CPU, NVIDIA, AMD, Apple, even on a Raspberry Pi
- No Python dependency: single binary
- GGUF support: the most published quant format in the open ecosystem
- Reasonable throughput: not the fastest, but within 30% of vLLM on single-GPU workloads
vLLM and TensorRT-LLM are stubbed in the Engines card for future implementation — they win on throughput at the cost of complexity (Python, much heavier install).
How to install
- Open Dashboard → Maintain → Runtime.
- Click Install runtime.
- Blueprint downloads the right build (~150-300 MB depending on OS), extracts it, and verifies the binary launches.
The install is idempotent — running it again upgrades to the latest release if your version is behind.
Verifying the install
After install, the Maintain tab's Runtime card shows:
- Installed version (e.g.
b4321) — matches llama.cpp's release tag - Latest available — what's on GitHub right now
- Binary path — for sanity-check via terminal
If installed === latest, you're current. If not, the Update
button pulls the new release.
Common pitfalls
- CUDA driver mismatch on Linux/Windows: the prebuilt CUDA llama builds expect CUDA 12.x runtime. If your driver is older, the binary loads but reports "no CUDA device" and falls back to CPU. Fix: update your NVIDIA driver, not the runtime.
- Antivirus quarantining the download: some EDR / corporate AV flags newly-downloaded EXEs. If the install fails silently on Windows, check your AV's quarantine folder.
- Disk space: the runtime is small (~300 MB) but each model you pull is multiple GB. Watch free space before pulling.
Need help with this in production?
Inspire AI Lab runs LLM optimization engagements end-to-end using Blueprint. If you'd rather hand the work to us instead of running it yourself, book a 30-minute review.