Semantic routing across a 3B + 70B pair

The cost of an LLM call scales (roughly) with the model size. A 70B response costs ~10× a 3B response in compute, latency, and dollars. If your application's traffic is mixed — some easy queries, some hard — sending everything to the 70B is wasteful. Semantic routing fixes that.

This walkthrough wires a 3B + 70B pair behind a router that picks per-request. The result: 75% of traffic served by the 3B, 25% by the 70B, with measured quality matching the 70B-everywhere baseline at ~5× lower average cost.

How the routing decision gets made

We use a small classifier model — fine-tuned BERT or a tiny LLM — that takes the user query as input and predicts difficulty. Two outputs:

• route: which model to use (small or large)
• confidence: how sure the router is

If confidence > 0.85, we honor the route. Otherwise we escalate to the large model. This guards against the router being wrong on edge cases — when in doubt, pay for the 70B.

Blueprint setup

In the Router card:

• Small model: Llama-3.2 3B Instruct Q4_K_M (running locally)
• Large model: Llama-3.3 70B Instruct Q4_K_M (running on a separate machine or cloud instance)
• Router model: a fine-tuned BERT-base classifier (we'll train this in a moment)
• Escalation threshold: 0.85 confidence
• Failure fallback: if the small model's response confidence is below 0.5, re-run on the large

Training the router

The router is a classifier on top of BERT-base. Training data: a few thousand labeled examples where each query has a binary label (easy / hard). We label by sending each query to both models, scoring both responses, and labeling "hard" when the small model's score is meaningfully lower than the large's.

In Blueprint's Train card:

• Model: bert-base-uncased
• Task: binary text classification
• Dataset: 4000 labeled query → {easy, hard} pairs
• Epochs: 3
• Batch size: 16

Training takes ~15 minutes on a 4090. The output is a small (~110MB) classifier that runs at ~5K queries/sec on CPU.

Wiring the routing

The Router card produces an OpenAI-compatible API on localhost:8090. Application code talks to this endpoint; the router decides where each request goes internally.

# Application code is unchanged from a single-model setup:
response = openai.chat.completions.create(
    base_url="http://localhost:8090/v1",
    model="auto",   # router picks
    messages=[{"role": "user", "content": query}],
)

# Under the hood:
# - Router classifier evaluates query → predicts (route, confidence)
# - If route=small AND confidence>0.85: send to 3B
# - If route=large OR confidence less than 0.85: send to 70B
# - Response streams back with a header X-Routed-To: small|large

Measuring

The Router card runs both the routed setup and a 70B-everywhere baseline against the same eval set, and reports:

About 5× cheaper per request with quality essentially flat. Latency drops by 2.5× because the small model is faster (and a much higher proportion of traffic hits it).

The hard queries

For curiosity, here are the categories of queries the router sent to the 70B in our test:

1. Multi-step reasoning — "Find the cheapest flight from A to B with a 4-hour layover in C." The small model's planning breaks down at 3+ steps.
2. Complex code generation — "Write a Python function that does X with these edge cases." The small model's code is buggier on non-trivial tasks.
3. Long-context retrieval — Queries that need to reference multiple parts of a 10K-token input. Small model loses track.
4. Math beyond arithmetic — Word problems with multiple constraints. Small models hallucinate.
5. Subtle classification — Distinguishing sarcasm, irony, intent. Small models miss tone.

This is roughly the 25-30% that justified the escalation. The other 70-75% — straightforward Q&A, simple summarization, easy classification, factual lookups — the 3B handles fine.

Failure modes we've seen

Router over-routes to large. Threshold too high or training data over-labeled "hard." Re-bench on a held-out set; if more than 35% goes to large, lower the confidence threshold or re-train the router with stricter "hard" labels.

Router under-routes (small model produces bad responses). Threshold too low or training data over-labeled "easy." If your quality is dropping, raise the threshold or add the failures to the router's training set.

Confidence calibration is bad. Some classifier models output uncalibrated probabilities — most predictions are 0.99 confident even when they shouldn't be. Train with temperature scaling or use a different base. Blueprint's Train card has a temperature-scaling step built in.

Drift over time. User behavior changes, the small model starts seeing queries it wasn't trained for. Re-train the router monthly with the latest production samples.

What this looks like cost-wise

At 1M queries/month:

Setup	Avg per req	Monthly
70B everywhere (API equivalent)	$0.0065	$6,500
Routed (75% 3B / 25% 70B)	$0.0013	$1,300
Routed + custom calibration on both	$0.0011	$1,100

Combine routing with custom calibration on the 70B and you knock another 15% off without quality loss. These optimizations stack.

What to try next

• Multi-tier routing: 1B / 7B / 70B with two thresholds
• Cost-aware routing: "Is this query worth the 70B call given the user's tier?"
• Caching at the route layer: cache router decisions for identical query prefixes

Each is a different overnight tweak. The pattern compounds across engagements; once you've shipped routing on one workload, the second is a fraction of the work.