Starting llama-server

What it does

The Start serving button in the Deploy tab runs:

llama-server \
  --model <path-to-gguf> \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size <CTX> \
  --n-gpu-layers <N> \
  --api-key blueprint-local \
  --metrics

It launches as a child process Blueprint supervises directly (separate from the Windows Service / systemd path — see Install service for the production-grade alternative). Output is captured into the deploy log stream; you can watch it update in real time on the Verify card.

The flags that matter

• --model — path to the GGUF. Blueprint resolves this from your model + quant pick.
• --ctx-size — maximum context length. Defaults to 4096. Larger = more VRAM. The Hardware tab's VRAM bar tells you the ceiling.
• --n-gpu-layers — how many transformer layers to put on GPU. Default is 999 (all). Set to 0 for CPU-only. Set to a number lower than the model's total layers for partial GPU offload — useful when the model is slightly too big for VRAM.
• --api-key — the bearer token clients send. Hardcoded blueprint-local for the in-app Verify chat; rotated to a random value when started via the Service card.
• --metrics — exposes a /metrics Prometheus endpoint the Dashboard's performance card scrapes.

How to start

1. Deploy tab → confirm the model is pulled (green check).
2. Start serving button.
3. The card transitions through starting → running as llama-server binds the port and reports ready.
4. Use the Verify chat below to send a test prompt and confirm the model responds.

Once running, the OpenAI-compatible API is at http://127.0.0.1:8080/v1/chat/completions with Authorization: Bearer blueprint-local. Any OpenAI client (Python openai, curl, the LangChain ChatOpenAI adapter, etc.) works against it unmodified.

When to use this vs the Service card

Use Deploy → Start serving	Use Service card
Interactive dev, demos	Production / always-on
You want to see logs live	You want auto-restart on crash
You'll close the GUI when done	You want the model to survive reboots

The Service card runs the same llama-server binary the same way, just supervised by the OS (systemd / Windows SCM) instead of the GUI process. See Install service for the prod path.

Common pitfalls

• Port 8080 already in use: another process (Jenkins, Tomcat, another LLM server) is squatting. Either kill that process or hand llama-server a different port via the advanced settings.
• OOM at startup: model + KV cache exceed VRAM. Reduce --n-gpu-layers (CPU offload some), reduce --ctx-size, or drop to a smaller quant.
• Quiet hang: llama-server is loading a multi-GB file from disk; cold start on slow disk can be 30-60 seconds. Watch the log lines — if they're flowing, it's working.