Evaluating quants

What the eval measures

For each candidate quant (your custom calibrated Q4_K_M, the stock Q4_K_M, your IQ4_XS, etc.) the eval harness runs every prompt in your eval.jsonl and records:

• Quality — the response's score against the expected output, using either exact match (normalized literal) or ROUGE-L F1 (token- level LCS overlap). Pick the scoring per-eval-row in the eval file or set a run-level default.
• Throughput — tokens-per-second during the generation phase.
• Time-to-first-token (TTFT) — wall-clock from request hitting the server to the first generated token. P50 and P95.
• Per-prompt breakdown — kept in the results JSON so a future drill-down surface can show you exactly which prompts moved when you switched quants.

The output lands in <runDir>/eval-results.json, rendered as a sortable table in the Eval card.

Why you need a held-out eval set

The imatrix calibration prompts and the eval prompts must be DIFFERENT. Otherwise the eval is measuring how well the quant overfits to its calibration set — a meaningless number.

Typical split:

• 70% calibration — drives the imatrix
• 30% eval — measures the quant's quality

If you only have a small total prompt budget (say, 100 prompts), use all of them for calibration and pull a separate eval set from a different time window of production data.

The eval set format

A .jsonl file, one JSON object per line:

{"prompt": "Classify the intent of: ...", "expected": "billing", "scoring": "exact"}
{"prompt": "Summarize in 1 sentence: ...", "expected": "User wants ...", "scoring": "rouge-l"}

Required: prompt, expected. Optional: scoring (defaults to the run-level setting).

How to run

1. Calibrate → open the run.
2. Step 5 — Eval. Upload your eval.jsonl.
3. Pick the candidate quants to compare. Usually: your calibrated one(s) + the stock GGUF (already on disk if you previously pulled it).
4. Set max tokens per response and the eval timeout per prompt.
5. Click Run eval. Each candidate is loaded into llama-server sequentially (eval-then-unload). Total time roughly: num_prompts * 5-30 sec * num_candidates.

The Eval card shows progress per candidate and a results summary on completion.

Reading the results

The summary table sorts by mean quality score by default. For each candidate you see:

• Quality (mean) — the headline number. Custom calibration should beat stock at the same file size.
• TTFT P50 / P95 — important for latency-sensitive serving.
• Tokens/sec (P50) — important for throughput-sensitive serving.
• File size — the actual byte count on disk.

The Best per workload badge appears on the candidate with the highest mean quality. That's usually but not always the one you want — if it's also the slowest, weigh against the latency you can tolerate.

Common pitfalls

• Tiny eval set (< 30 prompts): the per-candidate quality numbers have huge confidence intervals. Aim for 100+ prompts.
• Eval expects deterministic outputs but model temp > 0: a ROUGE-L eval on a temp=0.7 model is noisy. Either run eval at temp = 0 or use a scoring metric that tolerates variation.
• Comparing across runtimes: a quant evaluated under llama.cpp and a quant evaluated under vLLM will produce different numbers because the runtimes' tokenizers + sampling can differ. Keep the runtime fixed across the comparison.