llama-server metrics

Where the numbers come from

llama-server exposes a /metrics endpoint in Prometheus text format when launched with --metrics. Blueprint scrapes it every few seconds and renders the values on the Performance card.

The metrics live across two scopes:

• Lifetime counters — totals since the server started (e.g. total tokens generated)
• Live gauges — current state at scrape time (e.g. how many requests are processing right now)

Both are useful for different things. The card shows them in two groups.

The numbers that matter

Tokens-per-second (generation) — the headline throughput number. For a 7B Q4 on a 4090, expect 80-150 tok/s. For a 70B Q4 on the same card with CPU offload, expect 5-15. If you're significantly below these ranges, something's wrong (CPU offload too aggressive, batch size too small, KV cache misconfigured).

Time-to-first-token (TTFT) — wall-clock from request hitting the server to first generated token. Dominated by prefill time on long prompts. P50 is your typical user experience; P95 is your worst-case. A TTFT of 200ms on short prompts is healthy; 5+ seconds on long prompts is usually fine for non-interactive use.

KV cache usage ratio — what fraction of the allocated KV cache is currently in use. Sustained > 90% means you're about to evict; incoming requests will pay prefill costs they could have avoided.

Requests processing — concurrent in-flight requests. Compare against --parallel-slots (the cap). If processing == max and deferred > 0, you're capacity-constrained.

Requests deferred — requests waiting for a slot. Should be near zero in healthy serving. If it's climbing, raise --parallel-slots (if you have VRAM headroom) or scale out.

Reading the histograms

When the metrics card has been collecting for a few minutes, the TTFT and tokens/sec values stabilize into meaningful P50 / P95 percentiles. Watch:

• P50 TTFT spike on traffic with no obvious change → cache eviction or the supervisor just restarted llama-server
• P95 TTFT divergence from P50 → some prompts are dramatically longer than typical. Check whether you have a small set of outlier-long prompts dragging the tail.
• Tokens/sec drop with stable concurrency → KV cache pressure, GPU thermal throttling, or someone pulled a 30 GB model in the background and your I/O saturated.

Optimizing using these numbers

Symptom	Tweak
TTFT too high on long prompts	Enable prompt cache, or pre-warm with a representative prompt at startup
TTFT fine, tokens/sec low	Check GPU utilization — if it's low, batch size or parallel slots are bottlenecks
High tokens/sec, deferred > 0	Raise --parallel-slots (more VRAM needed)
KV cache always full	Reduce --ctx-size, or scale to a bigger GPU
Numbers all fine, app feels slow	Bottleneck is downstream of llama-server (network, your app, the user's browser)

Common pitfalls

• --metrics not enabled: the card shows "no data" because llama-server isn't exposing the endpoint. Edit the serve config and check the box.
• Tokens/sec looks like 0 when the server is busy: the metric is a counter, not a rate. Blueprint computes the rate by diffing scrapes. The first 30 seconds of any session will show stable zero until the diff window has data.
• Comparing to "vendor benchmark": NVIDIA / vendor numbers usually assume warm cache, batched workload, optimal sequence length. Your real numbers will be lower. The card's numbers are ground truth.