Building a calibration corpus from your client's prompts

The bottleneck on custom quantization isn't the calibration tooling — that's a one-liner. The bottleneck is the corpus you feed in. A garbage corpus produces a garbage imatrix produces a garbage custom quant. This post walks through how we build a calibration corpus on engagements where the client has production logs but isn't sure how to sample from them.

What "representative" actually means

A representative calibration corpus is one where the distribution of activations during the calibration forward pass matches the distribution of activations the model will see in production. That phrasing is precise but unactionable. The actionable version is: the corpus should contain prompts that look like, in roughly the same proportions, what your users will send.

"Look like" includes:

• Vocabulary and tokenization patterns — Code uses different tokens than prose. Legal language uses different tokens than support transcripts.
• Sequence length distribution — Short single-turn lookups vs. long multi-turn conversations.
• Prompt structure — Instruction format, system prompts, few-shot examples.

If your production hits 80% short factual lookups and 20% long instruction-following, your corpus should match that ratio. Not exactly — the imatrix is forgiving — but within a factor of 2.

Source: production logs

Most engagements have logs. The hard part is sampling from them.

A naive "grab the last 2000 prompts" almost never works. Production traffic is bursty and unbalanced — a single bad weekend of one customer hammering the API can dominate the sample. Worse, recent prompts may not represent steady state if the product is changing.

Our standard sampling shape:

1. Stratify by query type. If the client tags queries (intent classification, complaint, lookup, etc.), sample proportionally from each type. If they don't, cluster prompts via TF-IDF + k-means into 8-16 buckets and sample proportionally from each.

2. Stratify by length. Bucket prompts into short (less than 256 tokens), medium (256-1024), long (1024+). Sample with the same proportion as production traffic.

3. Stratify by time. Last week, last month, last 3 months. Catches drift in user behavior.

4. De-duplicate aggressively. MinHash with (num_perm=128, threshold=0.7) removes near-duplicates without erasing legitimate variation.

Code we use:

from datasketch import MinHash, MinHashLSH

lsh = MinHashLSH(threshold=0.7, num_perm=128)
kept = []
for i, prompt in enumerate(prompts):
    m = MinHash(num_perm=128)
    for token in prompt.split():
        m.update(token.encode("utf8"))
    if not lsh.query(m):
        lsh.insert(str(i), m)
        kept.append(prompt)

After dedup we typically have 30-60% of the original count. Then we down-sample to the target size (2000 by default).

PII handling

Production logs contain PII. Calibration corpora are stored alongside model artifacts and may be checked into source control. PII in the corpus is a leak waiting to happen.

We strip PII before the corpus leaves the client's environment:

• Named entities via spaCy's en_core_web_lg NER → replace PERSON with [NAME], ORG with [ORG], GPE with [CITY], etc.
• Email addresses, phone numbers, SSN-shapes, credit card patterns via regex → replace with format-preserving tokens like [EMAIL], [PHONE]
• Free-form numeric IDs (account numbers, order numbers) via length-and-context heuristic → [ID]

The replacement tokens preserve the structure of the prompt — the model still sees a prompt-shaped sequence — without leaking the actual identifiers. The imatrix doesn't care what specific names or numbers were there; it cares about the activation pattern, which is preserved.

We always have the client review a sample of post-stripped prompts before exporting. PII slips happen; a 100-prompt manual review catches them.

Don't include test set

Calibration is not training, but it's still an information leak. If the model sees prompt X during calibration and prompt X during evaluation, the importance scores will be unreasonably good for X-like prompts. Your eval will overstate the model's real-world quality.

We split production logs into three:

• Calibration set: 2000 prompts, used for imatrix
• Eval set: 500-1000 prompts, used for measuring before/after accuracy
• Holdout: everything else, kept aside for future validation

The split is by row (random with a fixed seed). MinHash dedup runs after the split so near-duplicates don't end up split across sets.

Format

The actual calibration corpus is a single .txt file with sequences separated by the model's end-of-text token:

<|endoftext|>
SELECT user_id, COUNT(*) FROM orders WHERE created_at > '2024-01-01' GROUP BY user_id;
<|endoftext|>
How do I cancel my subscription? I bought it last month and now I can't find the option.
<|endoftext|>
...

The separator matters — without it, llama-imatrix treats the whole file as one giant sequence and the importance scores get smeared. With it, each prompt is a discrete calibration unit.

For some models the separator is </s> or <|im_end|> instead of <|endoftext|>. Check the tokenizer's eos_token and use whichever the model expects.

Target size

We default to 2000 prompts at ~512-2048 tokens each. That's:

• Big enough that the imatrix has a stable signal
• Small enough to calibrate in 1-3 hours on a single 24GB GPU
• Diverse enough to avoid over-fitting to a narrow slice of production

For very large clients (millions of queries per day with high variance) we go to 5000. For tight, focused workloads (a single SQL prompt template) we drop to 1000 — there's less to capture.

Below 500 the imatrix is noisy and the resulting quant is unstable. Above 5000 we've seen diminishing returns on a half-dozen engagements and a meaningfully slower calibration run.

Validating the corpus

Before running the calibration, do these sanity checks:

1. Token count distribution. Plot sequence lengths. Should look like a long-tailed distribution similar to production, not a uniform spike.
2. Vocabulary coverage. What fraction of the model's tokens appear at least once in the corpus? Should be more than 40% for a 7B model; less than 20% suggests the corpus is too narrow.
3. Dedup ratio. What fraction of the raw prompts survived MinHash? If less than 10%, your logs were extremely repetitive (might be fine) or your threshold was too aggressive. If more than 90%, you didn't dedup much (probably fine).
4. Eyeball 50 random prompts. Read them. Do they look like the application? Catch garbled artifacts, encoding issues, prompts that are obviously test queries from a developer.

These take 30 minutes and have caught real problems on every engagement.

Common mistakes

Including the system prompt 10000 times. If your application's system prompt is "You are a helpful assistant" and every production query gets it prepended, your calibration data will be 90% identical system prompt. The imatrix will skew toward the assistant tokens. Strip the system prompt from the corpus (it's the same on every request anyway; the model has already seen it at attention) and include only the user-supplied portion.

Sampling from a single time period. Last week's prompts don't represent steady state if the product changed. Sample across at least a month.

Skipping PII review. "We'll strip it via regex" is famous last words. Always have the client read a sample.

Calibrating on the test set. Always re-stating this because we've watched smart teams do it.

What this looks like in practice

Once you've done this on three or four engagements, the workflow takes about half a day:

• 1 hour: pull and stratify logs with the client
• 30 min: dedupe + length filter
• 1 hour: PII strip + client review
• 1 hour: format + token-count sanity check
• 1-3 hours: run llama-imatrix + llama-quantize
• 30 min: measure before/after on the held-out eval set

By the end of the day, the client has a custom-calibrated GGUF that beats off-the-shelf by some measurable margin on their workload, and a calibration corpus they can re-use the next time they want to update the base model.