Semantic router

What it does

The router sits in front of two models you've registered: a small one (fast, cheap) and a large one (slow, expensive, smart). For each incoming query, it decides whether the small model can handle it alone or whether the request should escalate to the large model.

The decision uses two signals:

Semantic classification — a small embedding model scores the query against your "small handles this" examples. High score → small model. Low score → escalate.
Confidence threshold — after the small model generates a reply, the router can check the output's perplexity or self-reported confidence. Below threshold → escalate to the large model with the original query.

Production traffic for most apps splits roughly 60–80% queries the small model handles fine, 20–40% that need the large one. The average cost per query collapses to a fraction of "always-large" without the ceiling on hard queries that "always-small" would impose.

When to use it

The router pays off when two conditions both hold:

You're paying for inference (GPU time on your hardware, or per-token on an API) and the bill is non-trivial.
Your traffic has a heavy easy-tail — FAQ deflection, intent classification, "is this email spam," routine SQL generation, etc. If every query is a 50-page contract analysis, there's no easy tail to skim off.

If you're already running cheap and your traffic is uniformly hard, skip the router; it adds latency (the embedding classification step) without saving anything.

How to set it up

Pull both models via Deploy → Pull model. Common pairs: qwen-2.5-3b-instruct (small) + llama-3.1-70b-instruct (large).
Open Dashboard → Router. The card lists detected models on disk; pick small and large.
Set the escalation threshold. Start at 0.6. Lower = more escalation (safer, more expensive). Higher = less escalation (cheaper, more small-model errors).
Optional: prefix skipping — if the query starts with one of the listed prefixes (e.g. "explain the architecture of"), skip the small model entirely and go straight to the large one. Cheap way to encode "I know this is hard."
Send traffic through /llama/v1/chat/completions — the router intercepts and dispatches.

Tuning the threshold

The Router card shows three running counters: small-only, escalated, prefix-skipped.

Escalation rate < 10%: threshold is probably too low. You're losing speed to the small model on queries the small model is silently fumbling. Try raising to 0.65–0.7.
Escalation rate > 50%: threshold is too high. You're paying large-model cost on queries the small model would handle. Drop to 0.5–0.55.
The sweet spot for most workloads is 20–35% escalation.

Common pitfalls

Model mismatch: a 3B model can't escalate-recover from a 70B's failure modes. Pick small + large from the same family (Qwen 3B + Qwen 32B, Llama 8B + Llama 70B) so the failure boundaries are predictable.
No baseline measured: people enable routing without ever measuring what the large-only cost was. Spend a day running large-only first; record cost per 1k queries. Then turn routing on and compare.
The router IS a model: the embedding classifier loads ~80MB into RAM and adds ~5–15ms per request. On low-traffic apps this is fine; on a high-RPS endpoint factor it into your capacity planning.

Where escalation goes wrong

If you see suspicious answers in production, audit the escalated log: did the small model get it wrong AND not escalate? That means your threshold is too aggressive. Lower it and re-test on a held-out eval set before pushing the new value.