Calibrating Qwen 2.5 7B for SQL generation

This walkthrough takes a real workload — SQL generation from natural-language questions — and produces a custom-calibrated Qwen 2.5 7B Q4_K_M that outperforms the off-the-shelf bartowski version on the same eval set. We do it end to end in Blueprint without leaving the GUI.

What you need

• Blueprint installed and the runtime in place (Plan → Hardware → Deploy if you haven't already)
• ~3 hours of time on a 24GB+ GPU (most of which is the calibration run, in the background)
• A directory of SQL prompts and reference queries — we use Spider as a public stand-in for what you'd assemble from a real client's logs

Step 1: Get the base model

In the Plan tab, search for "Qwen 2.5 7B Instruct" and click Pull. Blueprint downloads the full-precision GGUF (about 15GB).

We're working with the unquantized base because calibration needs the full model to compute the importance matrix. You'll only need this once per base model — once the imatrix is computed and the custom Q4_K_M is built, you can delete the f16 to reclaim disk.

Step 2: Stage the calibration corpus

The corpus we feed to llama-imatrix should look like production. For SQL generation, that means natural-language questions paired with database schemas — what your users will actually send.

Blueprint's Calibrate tab has a corpus inspector. Point it at your prompts directory:

prompts/
  spider-train-sample.txt  # 2000 questions stratified across difficulty

The inspector shows:

• Token-length distribution (we want a long-tailed shape, not a uniform spike)
• Dedup ratio after MinHash (we want 30-60% survival)
• PII detection pre-flight (zero hits is good)

If the corpus looks reasonable, click "Use as calibration data."

Step 3: Build the imatrix

In Calibrate, pick the base model (Qwen 2.5 7B Instruct f16), pick the corpus, hit Start imatrix run.

What happens under the hood:

llama-imatrix \
  -m models/Qwen2.5-7B-Instruct.f16.gguf \
  -f prompts/spider-train-sample.txt \
  -o matrices/qwen-7b-sql.imatrix \
  --chunks 1000 \
  --ctx-size 2048

You don't have to type any of this — the Calibrate tab runs it for you with the right flags. Progress and tokens-processed/sec stream live in the panel.

On a 4090 with the Spider sample (2000 prompts, ~500 tokens each), this takes ~70 minutes. Coffee + email run.

Step 4: Quantize with the imatrix

When the imatrix run finishes, the Calibrate tab offers a "Build calibrated quant" panel. Pick the target quant (we want Q4_K_M for this exercise; for production you might compare Q4_K_M vs Q5_K_M).

The quantizer runs:

llama-quantize \
  --imatrix matrices/qwen-7b-sql.imatrix \
  models/Qwen2.5-7B-Instruct.f16.gguf \
  models/Qwen2.5-7B-Instruct-Q4_K_M-sql.gguf \
  Q4_K_M

Quantization takes about 5 minutes. The output is a 4.4GB GGUF that loads in llama.cpp identically to the off-the-shelf quant.

Step 5: Evaluate

Calibrate's Eval panel runs your model against a held-out test set and reports the score. We load both the off-the-shelf quant (downloaded separately from bartowski) and the custom one, and compare.

For SQL generation, the eval metric is execution accuracy: does the generated SQL produce the same result rows as the reference SQL when run against the schema? Spider provides a graded eval harness; Blueprint's Eval panel wraps it.

Our results on a 500-question held-out subset of Spider:

The custom Q4 beats the stock by 9.2 percentage points on the workload we care about. General capability (MMLU) drops by 0.5 — within noise, well within what we accept.

Step 6: Deploy

Back in Plan / Deploy, point the runtime at the custom GGUF. It serves through the same engine as before. The OpenAI-compatible API on 127.0.0.1:8080 is unchanged.

For Spider-shaped queries, every chat completion now uses the custom-calibrated model. For an internal SQL-helper application, that's the entire deployment.

What this saves an engagement

Before Blueprint, this workflow was:

1. SSH into a workstation
2. Clone llama.cpp
3. Compile with the right CUDA flags
4. Pull the f16 GGUF (figure out the download command)
5. Write a shell script to build the imatrix
6. Wait, hope it didn't OOM
7. Write a second script to quantize
8. Stand up llama-server, point at the new GGUF
9. Write a third script to run the eval harness
10. Manually compare numbers

About a day. With Blueprint, the same workflow is the three or four button clicks above. The first time you do it, you spend 20 minutes navigating the UI. The fifth time, it's a 5-minute task.

What to try next

• Compare custom Q4_K_M against custom Q5_K_M (~5.0GB, marginally higher accuracy)
• Run the same calibration with a larger / more diverse corpus (5000 prompts vs 2000)
• Apply the same flow to a different domain (try contract review with CUAD prompts)

Each one is the same Blueprint panel, different inputs. The pattern compounds; once you've done it twice, the engagement-level repeatability is real.

Sample data

Spider's sample we used in this walkthrough is publicly available from Yale. For a real engagement you'd build the equivalent corpus from your client's production logs (we cover the methodology in Building a calibration corpus).