Calibrated quantization

What this step does

You've calibrated an imatrix on prompts that look like your workload (see Custom imatrix calibration). Now you turn that imatrix into one or more quantized GGUFs at the bit widths you actually want to use.

Blueprint runs llama-quantize with --imatrix <run>/imatrix.dat and outputs to <run>/quants/<target>.gguf for each target you pick.

Picking the targets

The quant naming follows llama.cpp's convention:

• Q4_K_M — the standard 4-bit. Best quality-per-byte; the default for most production deployments.
• Q4_K_S — smaller 4-bit. Worse quality, slightly smaller file. Rarely worth picking over Q4_K_M.
• Q5_K_M — 5-bit. Higher quality than Q4 at the cost of ~25% more file size.
• IQ4_XS — newer 4-bit "I-quant." Smaller than Q4_K_M, often comparable quality with a good imatrix. Worth testing.
• IQ3_M / IQ3_S — 3-bit. Significant quality loss without careful calibration. Only ship these if your eval validates they hold up.
• Q8_0 — 8-bit. Near-lossless. Use when accuracy matters more than VRAM (rarely).

When in doubt, pick Q4_K_M and IQ4_XS — both with your imatrix — and let the eval step tell you which one ships.

How to run

1. Calibrate → open the run (must be past Step 3 with a successful imatrix).
2. Step 4 — Calibrated quantization.
3. Tick the target quants you want.
4. Click Run quantization. Each target takes a few minutes depending on model size and your CPU (quantization is CPU-bound, not GPU). A 70B model can take 30-60 min per quant.
5. Output GGUFs land at <runDir>/quants/<target>.gguf and are immediately visible in the Eval step's candidate picker.

Why this isn't automatic with the imatrix step

Two reasons it's a separate explicit step:

1. Picking targets requires judgment. Different deployments want different bit widths; we don't pre-quant every possible target because it would waste hours of CPU per run.
2. Quantization is CPU-intensive. A 70B at Q4 is a 30-60 min job on a fast desktop. Letting the user choose which targets to spin on means they can run only what they'll actually eval.

Common pitfalls

• Running quantization on a slow CPU: a 70B Q4_K_M on a low-power laptop can take hours. Run on your fastest box.
• Forgetting to re-quantize after re-calibration: if you re-ran the imatrix on fresh data, the existing quants are stale. Delete the old <run>/quants/*.gguf files and re-run this step.
• Quantizing without a calibrated imatrix: technically possible but defeats the whole point. If you skipped Step 3, the resulting GGUF is no better than what's already on HuggingFace.