What imatrix actually does (and why off-the-shelf Q4 leaves accuracy on the table)

When you download Qwen2.5-7B-Instruct-Q4_K_M.gguf from bartowski's HuggingFace, you're getting a model whose 4-bit quantization was calibrated against the WikiText and C4 corpora. That's fine if your application looks like Wikipedia or web crawl. It's leaving accuracy on the table if your application looks like anything else — SQL queries, contract clauses, support transcripts, code completions.

This is what imatrix (importance matrix) calibration fixes, and why we re-calibrate quants on every consulting engagement that ships a deployed model.

The compression problem in one paragraph

A 7-billion-parameter model in float16 is 14GB. At Q4_K_M it's about 4.4GB — a 3.2× reduction. That compression comes from reducing each weight from 16 bits to roughly 4.5 bits on average, using a clever per-block scheme that allocates more bits to weights that matter more. But "matter more" is defined relative to some input distribution. If that distribution doesn't match yours, the bit allocation is suboptimal for your workload.

What imatrix is, mechanically

llama-imatrix runs a forward pass of the unquantized model over a calibration corpus and records, for each weight matrix, the magnitude of activations that hit each row/column. The result is a per-weight importance score: "this weight column gets multiplied by big numbers when this kind of input arrives, so it'd better be quantized accurately."

The output is a .imatrix file (a few MB) that you feed into llama-quantize to produce a calibrated GGUF. The quantizer uses the importance scores to allocate higher precision to weights that empirically matter and lower precision to weights that don't.

The trick: the importance scores depend on the calibration corpus. Different corpora yield different scores, which yield different quantizations.

Why the default calibration isn't great for your workload

bartowski's quants — which is what most people use — are calibrated against groups_merged.txt, a hand-curated mix of WikiText, code, multilingual text, and conversational data. It's a reasonable general-purpose blend. It is not your blend.

A real example from a recent engagement. Client had a 7B model serving SQL generation queries. The base Q4_K_M (bartowski's calibration) scored 64% on their held-out test set. We re-calibrated against 2000 of their actual prompts (anonymized) and re-quantized. Same model, same quant level, same disk size. New score: 73%.

9 percentage points. Free, in the sense that it costs nothing at inference. The only cost is one ~2-hour calibration run during the engagement.

Building a calibration corpus

The calibration corpus should look like the inputs your deployed model will see. Not exact copies — that would overfit — but representative samples from the same distribution.

Practical rules:

• 2000-5000 sequences is plenty. Diminishing returns above that. Below 500 you risk over-fitting calibration to a narrow set of prompts.
• Each sequence should be ~512-2048 tokens. Too short and the importance signal is noisy; too long and the calibration run takes forever.
• Mix difficulty. Half should be representative average cases. A quarter should be common-mode failure scenarios. A quarter should be edge cases your eval set cares about.
• Don't include test set examples. This is calibration data, not training data, but you still want a clean split.

We typically build the corpus by:

1. Pulling 5000 anonymized prompts from the client's logs (with their consent)
2. Stripping PII
3. Deduplicating by minhash similarity
4. Down-sampling to 2000 with stratified sampling across query types

The whole corpus lives in a single .txt file with sequences separated by <|endoftext|>.

Running the calibration

llama.cpp ships the tooling:

# Step 1: produce the imatrix from the unquantized base.
llama-imatrix \
  -m models/Qwen2.5-7B-Instruct.f16.gguf \
  -f calibration-corpus.txt \
  -o models/Qwen2.5-7B-Instruct-custom.imatrix \
  --chunks 1000 \
  --ctx-size 2048

# Step 2: quantize with the imatrix.
llama-quantize \
  --imatrix models/Qwen2.5-7B-Instruct-custom.imatrix \
  models/Qwen2.5-7B-Instruct.f16.gguf \
  models/Qwen2.5-7B-Instruct-Q4_K_M-custom.gguf \
  Q4_K_M

The calibration run on a single 24GB GPU takes 1-3 hours for a 7B model and a 2000-sequence corpus. The actual quantization is fast (a few minutes).

The output GGUF is byte-for-byte the same shape as the off-the-shelf one. It loads in llama.cpp without any flags or special handling. The only thing different is which weights got high-bit treatment.

What to measure

The accuracy lift should be measurable on your held-out eval set. We always insist on three measurements:

1. Your domain eval, before and after. This is the headline number — the one we report to the client.
2. A general-capability eval (HellaSwag, MMLU, or a small held-out set of out-of-domain prompts). Catches the case where custom calibration over-fit your domain and broke generality.
3. Perplexity on your calibration corpus. Should drop noticeably. If it doesn't, the imatrix computation went wrong.

A reasonable shape for a successful custom calibration:

Metric	Default Q4	Custom Q4	Delta
Domain eval	64%	73%	+9pp
MMLU	56%	55%	-1pp
Domain perplexity	4.8	3.9	-19%

The general MMLU drop of 1pp is the price you pay. If it drops more than 3pp, your calibration corpus was too narrow — add more diverse data and re-run.

When custom calibration doesn't help much

A few cases where custom imatrix isn't worth the engagement effort:

Your workload genuinely is generic English text. General chat assistants, broad summarization. The default calibration was built for this. Expect maybe 1-2pp lift, not 9.

You're running at Q8 or higher. Q8 quantization is already accurate enough that there's little headroom for calibration to recover. The bigger wins are at Q4 and below.

Your eval set is tiny. With less than 500 eval examples, the noise floor is large enough that 9pp shifts could be measurement error. Build a bigger eval set first, then re-calibrate.

Your task is far from the base. Custom calibration helps quantization match the distribution; it doesn't help if the base model fundamentally doesn't know your domain. Use LoRA / SFT for that, then re-calibrate the fine-tuned model.

What this looks like in practice

The calibration step has become standard in every deployment engagement we run. It's cheap, repeatable, and the lift compounds with whatever other optimizations are happening downstream (LoRA, prompt compression, routing). The accuracy floor of "what a custom-calibrated Q4 can deliver" sits noticeably higher than "what bartowski's Q4 delivers," and once you've internalized that, it's hard to justify shipping the default.