Verifying a deploy

What it tests

The Verify chat below the Deploy → Start serving button is a thin chat UI that points at the locally-served llama-server (the one the Start action just spawned). It tests, in order:

1. The server is reachable — bearer token works, port responds
2. The model loaded — first token comes back within a reasonable time (a 70B Q4 cold-start can take 30s; a 7B Q4 should be sub- second)
3. Outputs look sane — for the model you picked, the response should be coherent in the expected language, follow the prompt's instruction, and stop at a reasonable length

How to use it

1. Once the server status reads running, the Verify card unlocks.
2. Type any prompt. Useful smoke-test prompts:
   • "Write a haiku about quantization." — checks creative + format
   • "What is 47 × 53?" — checks arithmetic + reasoning
   • "Return a JSON object: {city: 'Paris', country: 'France'}." — checks instruction-following + structure
3. Watch the response stream token-by-token.
4. Check the metrics: tokens/sec, time-to-first-token, total tokens. If these look bad relative to your hardware's expectations, see the LLM metrics article.

What "looks wrong" looks like

• Garbage / unicode noise / repeating tokens: the model is loading with the wrong tokenizer or a corrupted GGUF. Re-pull the model.
• Refuses every prompt: the chat template the GGUF embedded doesn't match what llama.cpp expected. Check the model's HuggingFace card for the right chat template; some models need --chat-template chatml or similar flag.
• Quiet hang on first token, then sudden flood: prefill is slow but generation is fine. Expected on long prompts + small GPUs — consider enabling the prompt cache if you'll see this prompt again.
• Crashes mid-response with OOM: KV cache outgrew available VRAM. Restart with a smaller --ctx-size or move to a bigger GPU.

What it doesn't test

Verify is a smoke test, not an eval. It tells you the server is alive and roughly sane; it doesn't tell you whether the model is better than an alternative for your workload. For that, run the Calibration eval pipeline against your actual prompts.

Common pitfalls

• "Works in Verify, fails in my app": Verify uses Authorization: Bearer blueprint-local and base URL http://127.0.0.1:8080. If your app uses different creds or hits via a different network path (Docker, WSL2, remote machine), the request might never reach the server.
• One-prompt test isn't enough: a model that nails "write a haiku" might blow up on your real workload. Test with prompts that look like production traffic.