← All articles · Calibrate
Custom imatrix calibration
Calibrate a quantization against your own prompts so the bits land where YOUR workload pushes activation magnitude — typically 8-15% accuracy gain at the same file size.
3 min read
What an imatrix is
An imatrix ("importance matrix") is a per-layer record of which activations have high magnitude during the model's forward pass on some input. Quantization is destructive — you can't keep all the bits, you have to throw some away — and an imatrix tells the quantizer where the model "thinks loudest" on data that looks like yours, so it can preserve those activations and discard others.
The off-the-shelf Q4 GGUFs published on HuggingFace are calibrated against the C4 dataset (Common Crawl, broad web text). That's fine if your workload is broad web text. It's noticeably suboptimal if your workload is customer-support transcripts, SQL prompts, legal contracts, code, or any narrow domain.
How much does it help
Empirically, on a 200-prompt eval set drawn from the target workload, a custom Q4_K_M imatrix calibration produces a quant that scores 8-15% higher than the stock Q4_K_M on that eval set — at exactly the same file size. The trade-off: 1-3 hours of one-time calibration GPU time per (model, quant).
If you're shipping a self-hosted LLM into production, custom calibration is almost always worth it. The gain compounds with every subsequent prompt; the cost is once.
How to run it in Blueprint
- Calibrate → New run. Name the run after the workload (e.g.
q1-support-intent). - Step 1 — Upload prompts. Drop a .txt file containing ~200-500 prompts that look like your production traffic. One prompt per line or paragraph; total ~50k-200k tokens.
- Step 2 — Pick the base model + quant. The one you want to calibrate. The base GGUF must be pulled already (Deploy → Pull model).
- Step 3 — Run imatrix. Click. Takes ~30-90 min on a 4090, ~10-20
min on an H100, depending on prompt count + model size. The run
produces an
imatrix.datfile in the run's directory. - Step 4 — Calibrated quantization. Pick target quants (e.g.
Q4_K_M, IQ4_XS). Each one produces a
<run>/quants/<target>.gguffile. See Calibrated quantization. - Step 5 — Eval. Compare the new quant against the stock one on a held-out eval set. See Evaluating quants.
Picking the right prompts
The imatrix is only as good as the data driving it. Rules:
- Representative, not idealized. Use prompts that look like your actual production traffic, warts and all. Typos, half-sentences, weird formatting — keep them.
- Diverse enough. 50 near-identical prompts give a brittle imatrix. 500 prompts spanning the actual variation in your traffic give a robust one.
- Long enough. Aim for ~50k-200k tokens total. Less than 50k and the calibration is noisy; more than 200k and you're paying for diminishing returns.
Common pitfalls
- Calibrating on your eval set. The imatrix is "training" in a loose sense — using the same prompts for calibration AND eval gives you a number that's not predictive. Hold out a separate eval set.
- Calibrating on synthetic data. GPT-4-generated prompts won't look like your real customers' prompts. The calibration optimizes for GPT-4's style, not yours.
- Forgetting to re-calibrate when the workload drifts. If your
product expands from "customer support intent" to "intent + sentiment
- summarization," yesterday's imatrix is now mis-calibrated. Re-run every few months on fresh production data.
Need help with this in production?
Inspire AI Lab runs LLM optimization engagements end-to-end using Blueprint. If you'd rather hand the work to us instead of running it yourself, book a 30-minute review.