Custom Q4_K_M for a customer-support intent classifier

Customer support intent classification is one of the cleanest LLM workloads: short input (the customer's first message), small set of output labels (cancel, refund, billing question, technical issue, etc.), high volume, latency-sensitive. It's also a workload where Q4 quantization usually loses 5-8 percentage points compared to f16 — unless you re-calibrate.

This walkthrough takes 4000 support transcripts, re-calibrates a Qwen 2.5 7B Instruct Q4_K_M against them, and ships a custom GGUF that matches the f16 baseline on accuracy. Same disk size as stock Q4, ~3× smaller than f16, no quality loss on the trained task.

The corpus

We use a synthetic-but-representative corpus of 4000 anonymized customer support first-messages, mapped to 12 intent labels:

{"text": "I want to cancel my subscription, I bought it last week", "intent": "cancel"}
{"text": "My card was charged twice for the same order, can you refund one?", "intent": "refund_dispute"}
{"text": "The product I received is broken, who do I contact?", "intent": "product_defect"}

For a real engagement you'd assemble the equivalent from the client's support logs (we cover the methodology in Building a calibration corpus). PII strip, dedupe, stratify by intent. 4000 prompts is enough.

The calibration

In Blueprint's Calibrate card:

• Base model: Qwen 2.5 7B Instruct (full precision f16, 15GB)
• Calibration corpus: the 4000 anonymized support transcripts as a single .txt file
• Output: custom imatrix → custom Q4_K_M GGUF

The card runs:

llama-imatrix \
  -m models/Qwen2.5-7B-Instruct.f16.gguf \
  -f corpus/support-anon-4000.txt \
  -o matrices/qwen-support.imatrix \
  --chunks 1000 \
  --ctx-size 1024

llama-quantize \
  --imatrix matrices/qwen-support.imatrix \
  models/Qwen2.5-7B-Instruct.f16.gguf \
  models/Qwen2.5-7B-Instruct-Q4_K_M-support.gguf \
  Q4_K_M

Two clicks in the Blueprint UI. ~50 minutes total on a 4090, mostly the imatrix run.

Evaluation

Eval set: 500 held-out support transcripts, never seen during calibration. The metric is exact-match intent classification.

Three models go through the same eval:

1. f16 base — the unquantized starting point
2. Stock Q4_K_M (bartowski) — what you'd download from HuggingFace by default
3. Custom Q4_K_M — what we just built

The custom Q4 sits 0.8pp below the f16 baseline. The off-the-shelf Q4 sits 7.2pp below. The calibration recovered 88% of the f16-vs-Q4 gap — at no inference cost.

Why the gap

The bartowski Q4 was calibrated against general web text (C4 + WikiText). Support transcripts have a vocabulary that overlaps somewhat with general text but is biased — heavy on action verbs ("cancel," "refund," "broken"), product names, customer-emotion words. The default calibration didn't allocate bits well for these tokens; our custom calibration did.

Deploying

Point Blueprint's Plan → Deploy at the new GGUF. The OpenAI-compatible API on 127.0.0.1:8080 serves it. Application code is unchanged from a stock Q4 deployment.

For an intent-classification application, you'd typically wrap this in a simple system prompt:

You are a customer support intent classifier.
Given a customer message, output a single intent label from:
cancel, refund_dispute, product_defect, billing_question, technical_issue,
shipping_inquiry, account_access, feature_request, general_complaint,
positive_feedback, partner_inquiry, other

Customer message: [USER_MESSAGE_HERE]
Intent:

The model outputs one label. Latency on a 4090 with the custom Q4 is ~50ms per request. Throughput is ~120 requests/sec at concurrency 8.

What this saves at scale

For a deployment doing 100,000 support classifications per day:

Setup	Accuracy	Daily cost
GPT-4o-mini API	~91%	$30
Self-hosted Llama-3.3 70B (over-killed)	~92%	$50 amortized GPU
Self-hosted Qwen 2.5 7B Q4 (stock)	82%	$12 amortized GPU
Self-hosted Qwen 2.5 7B Q4 (custom-calibrated)	89%	$12 amortized GPU

The custom 7B Q4 sits within 2 percentage points of GPT-4o-mini at lower cost. The stock Q4 doesn't. The calibration step — one button click in Blueprint — is what makes the cheap setup competitive with the API.

What can go wrong

A few honest failure modes:

Corpus too narrow. If your 4000 transcripts are all from one product line, calibration over-fits and general capability drops. Mix in transcripts from across the business.

Imatrix run OOMs. Drop --ctx-size to 1024 (we did) or use --chunks 500 if you're tight on time.

Quantization succeeds but the model outputs garbage. Almost always a tokenizer or model architecture mismatch — make sure the f16 you calibrated against and the GGUF the quantizer produced match the same model. Blueprint enforces this; hand-rolled setups sometimes don't.

Accuracy went up on the trained intents but down on rare ones. Stratify the calibration corpus by intent. We pulled 333 examples per intent (×12 intents = 4000) to keep balance.

What to try next

• Compare custom Q4_K_M against custom Q5_K_S (closer to f16 quality, slightly bigger)
• Layer a LoRA fine-tune on top: with custom calibration recovering 88% of the Q4-vs-f16 gap, can a small LoRA close the rest?
• Apply the same flow to a different domain (sentiment classification, content moderation) — same pattern, different corpus

Each is a different Blueprint button click. The end-to-end engagement skill is "give me a workload, a corpus, and 2 hours, and I'll hand back a custom-quantized model that's better than the off-the-shelf at the same size."