Building an evaluation harness that survives prod

Every model deployment we've shipped has an evaluation harness. Half the value of a consulting engagement is the harness; the model itself is replaceable, the harness is the institutional memory.

A good harness answers three questions, in real time:

1. Is this model better or worse than the last one? (regression detection)
2. What's failing, and is the failure mode getting worse over time? (failure-mode tracking)
3. What does the user actually want, and are we delivering it? (alignment to value)

The questions sound simple. Building a harness that answers them honestly takes effort.

The four-layer eval pyramid

We split evaluation across four layers, from cheapest and most frequent to most expensive and least frequent.

Layer 1: Smoke tests (every commit)

A handful of fixed prompts (~30-100) with expected outputs. Runs on every deploy. Catches gross regressions — model outputting in the wrong language, format collapse, refusing every request, etc.

Implementation: a JSON file with {prompt, expected_pattern, must_not_contain} triples. Run all of them, fail the deploy if any fail.

def smoke_test(model, prompts):
    failures = []
    for case in prompts:
        out = model.generate(case["prompt"], max_tokens=200)
        if not re.search(case["expected_pattern"], out):
            failures.append((case["id"], "missing pattern"))
        for forbidden in case.get("must_not_contain", []):
            if forbidden in out:
                failures.append((case["id"], f"contained: {forbidden}"))
    return failures

Smoke tests run in less than 2 minutes. They don't replace deeper eval; they fail fast on the obvious.

Layer 2: Held-out eval set (every model change)

500-2000 examples drawn from production traffic, with held-out reference answers. Each example has:

• The prompt as the user sent it
• The expected answer (graded by humans or by GPT-4 against an explicit rubric)
• A category tag (which kind of query is this)
• A difficulty tag (easy / medium / hard)

This is the headline number — "we went from 73% to 81% on the held-out eval." It runs whenever the model, the prompt, or the retrieval pipeline changes.

The eval set is precious. Don't train on it, don't calibrate on it, don't pick prompts based on it. We use cryptographic separation: the eval set sits in a separate S3 bucket with a different access role than training data, and the harness is the only thing that reads from it.

Layer 3: Production scoring (continuous)

Live traffic, scored as it happens. Two flavors:

Self-scoring — the model rates its own confidence on each response. Cheap, noisy, useful as a flag.

LLM-as-judge — a stronger model (GPT-4 or Claude) scores a 1% sample of production responses against your rubric. We do this on every deployment for two weeks, then drop to 0.1% sampling for steady-state monitoring.

The signal you're looking for: distribution shift. If yesterday's score distribution was centered at 4.2 and today's is at 3.8, something changed. Drill in.

Layer 4: Human review (weekly)

A weekly review of 50-100 random production samples by a domain expert. This catches things automated scoring misses — subtle factual errors, tone problems, edge cases that weren't in the rubric.

Every human-flagged failure becomes a new eval example. The eval set grows by ~50 examples a week from this loop. After 6 months you have a 1500+ example eval set entirely composed of real production failures + their corrections. That's the institutional asset.

Metrics that matter

Generic metrics are usually bad. "Accuracy" requires a definition of correct, which is domain-specific. We pick metrics by application shape:

Classification / extraction. Precision / recall / F1 on the actual labels. Don't accept "looks correct" — define the schema and score against it.

Question answering with citations. Three numbers: (a) is the answer correct, (b) does it cite the right source, (c) does the citation actually support the claim. Two and three catch hallucinated citations.

Code generation. Pass rate on a held-out test suite. Don't grade by "does the code look right." Run it.

Summarization. Faithfulness (does the summary reflect the source) and coverage (does it include the important points). Both need rubric-based human or LLM-judge scoring.

Open-ended generation (creative writing, drafts). Pairwise comparison against a reference model. "Is this better than what GPT-4 produced for the same prompt?" The answer is more honest than a single absolute score.

Don't try to summarize a deployment with one number. We ship a dashboard with 4-8 metrics, one per query category. A model that's better on average but worse on the highest-value category isn't a win.

What we always include

A few elements every harness gets:

Versioning. Every eval run records the model version, prompt template version, retrieval pipeline version. Six months later you can answer "what changed between week 12 and week 13?"

Latency tracking. Accuracy without latency is half the story. We always include p50 / p95 / p99 latency and tokens-per-second per query category.

Cost tracking. Per-query token spend, broken down by prompt / completion / RAG context. Production costs that surprise leadership are an avoidable failure.

Failure-mode tagging. Every failed eval example gets a category tag: "hallucinated fact," "wrong format," "refused valid query," "incomplete answer." Tracking failure modes over time shows whether they're getting worse or better.

Adversarial examples. A small set (50-100) of known-tricky prompts. The model should handle all of them; if one regresses, the model has degraded somewhere subtle.

Anti-patterns we've seen

Vanity benchmarks. Reporting MMLU score for a customer-support model. Generic benchmarks are decorative; they don't predict production performance.

Training-set leakage. The held-out eval set somehow appears in the calibration corpus or the fine-tuning data. Always cryptographically separate.

Once-and-done eval. Running the harness at launch and never again. Models drift, prompts drift, user behavior drifts. Eval needs to be recurring.

Eval set capture by the loudest stakeholder. Whoever complains loudest about a failure gets their case added, eval gets skewed toward whatever they care about. Have a documented intake process.

Optimizing for the eval set. "We improved the harness score by 8 points" while production satisfaction dropped. If the eval is gameable in a way that disconnects from real value, fix the eval.

What this gives you

A working harness changes the engagement dynamic. Instead of "we think the new prompt is better" you have "the new prompt scores 81% vs the old 76% on n=1500 held-out examples, with no regression on any subcategory." Stakeholders accept the second; they argue about the first.

It also changes the conversation with the client. Six months in, the harness has 1500 examples and tracks 6 metrics across 8 query categories. When the client wants to swap models or try a new approach, the harness gives a 24-hour answer instead of a week-long debate.

That asset compounds. The model is the depreciating piece — it gets superseded by something newer every 6-12 months. The eval harness is the lasting one. We hand it over at the end of every engagement.