Prompt cache in production: what hits, what misses, and how to design around it

The economics of an LLM API call are bimodal. Prefill (processing the input prompt) is compute-dominated. Decode (generating the output) is memory-bandwidth-dominated. For typical chat traffic, prefill is roughly 2-5× the wall-clock cost of decode per request.

A prompt cache makes prefill cheap for the part of the prompt that doesn't change between requests. For applications with a stable system prompt and retrieved context, that's the majority of the prefill work. Hit rates of 60-90% on well-designed applications are normal; they push average request latency down by 50-80% and free GPU cycles for new work.

This post is about what makes a prompt cache hit, what makes it miss, and how to structure prompts so it hits more often.

How the cache works

Inside a transformer, the prefill phase computes attention keys and values (KV pairs) for every position in the input. A prompt cache stores these KV pairs keyed by a prefix hash. When a new request arrives whose prompt starts with the same prefix, the server skips re-computing the KV pairs for the shared portion and resumes from the divergence point.

In llama.cpp this is --cache-reuse (off by default) and a few related flags. In vLLM it's prefix caching (opt-in but well-supported). In TGI and TensorRT-LLM it's similarly available.

The key insight: the cache is keyed by token-id prefix, not by string prefix. "Hello, world!" and "Hello, world! " (with a trailing space) tokenize to different sequences and miss the cache. We've seen production systems lose 40% of their potential hit rate because some path in their code was adding a trailing newline.

What hits

A cache hit requires three things:

1. Same tokens, same order, at the very start of the prompt. Diverge at position N and the cache contributes 0 for positions ≥ N.
2. Same model + same context size + same quantization. The KV pairs are computed by the model; they're not portable across models.
3. Same sampling parameters that affect prefill. Temperature and top-p don't affect prefill (they only matter for sampling) so they don't break the cache. But --no-prefill flags, custom positional embedding scales, etc. do.

The most common cache hit shape is:

[stable system prompt][stable few-shot examples][stable retrieval context][user query]

The first three components are identical across requests for a given application. The user query changes. The cache covers everything before the user query.

What misses

Things that look like they should hit but don't:

Whitespace inconsistencies. Tab vs. spaces, trailing newlines, Windows vs. Unix line endings. We've seen production code paths that called .strip() on the system prompt in one place and not another. 40% hit-rate cliff.

Timestamps in the system prompt. "Today is Monday, June 26, 2026" embedded in the system prompt means every request has a different prefix. Move dynamic content to after the static portion of the prompt.

Per-user personalization in the prefix. Hello {NAME}, here's your context at the top of the prompt means every user gets their own cache entry. Move personalization to the end.

RAG context that re-orders chunks. If your retriever returns 5 chunks per query and the order is non-deterministic, every query is a miss. Sort by hash or by stable score before assembling the prompt.

Streaming vs. non-streaming. Some servers compute attention slightly differently for streaming requests. Mostly handled correctly in modern versions but worth verifying — a 100% miss rate when you flip the stream flag is a common bug.

Hit-rate measurement

Don't trust the docs; measure the rate. llama.cpp reports cache stats in /v1/internal/metrics (when --metrics is on) or directly to stdout with --log-cache. vLLM exposes prefix_cache_hit_rate in its Prometheus metrics. TGI is similar.

A reasonable shape for a well-designed application:

If you're below 50%, something is structurally wrong with how prompts are assembled. Below 20%, the cache is effectively off — review the misses one at a time.

Designing for hits

Three rules:

1. Sort everything dynamic to the end. Static system prompt, then static few-shot examples, then sorted retrieval context, then the user query last. Anything that varies per-request goes after everything that doesn't.

2. Canonicalize the prefix. A single function that produces the prefix as bytes, with deterministic whitespace, ordering, and Unicode normalization. Every request path goes through this function.

def canonical_prefix(system_prompt: str, examples: list[dict]) -> str:
    sys = system_prompt.strip().replace("\r\n", "\n")
    parts = [f"<|im_start|>system\n{sys}<|im_end|>\n"]
    for ex in sorted(examples, key=lambda e: e["id"]):
        parts.append(format_example(ex))
    return "".join(parts)

The sorted is critical. Inconsistent ordering of identical content kills hit rates.

3. Profile prefix divergence under load. Run a synthetic load test that submits 100 prompts with the same nominal system prompt and inspect the cache stats. If hit rate isn't 99%+, something is producing per-request variation in the supposedly-static prefix.

Cache eviction

Production caches have finite capacity. When the cache fills, the least-recently-used entries get evicted. Three strategies for managing this:

LRU is fine for chat-shaped traffic with a few distinct prefix variants. Most cache hits will be on the most-recently-used 1-2 prefixes anyway.

Explicit pinning for the highest-value prefixes. llama.cpp's --cache-pin (or vLLM's equivalent) lets you mark a prefix as never-evicted. Worth using for your top-2 system-prompt variants.

Sharded caches by tenant if you're multi-tenant. One cache per customer isolates eviction pressure. Bigger memory footprint, but predictable per-tenant performance.

The wrong move is to over-size the cache. Cache memory is GPU memory — every megabyte you give the cache is a megabyte you can't use for activations or longer batches. Start at 1GB of cache, measure hit rate, and grow only if you see thrashing.

What hits give you

The first-order effect is latency reduction — TTFT (time to first token) drops 60-80% on hits. The second-order effect is throughput: prefill cycles freed by the cache get spent on more decode tokens, so total tokens-per-second of the server goes up.

On a 4090 serving a 7B model with a 4K-token system prompt:

Setup	Avg TTFT	Tokens/sec/server
No cache	850 ms	580
75% hit rate	280 ms	920
95% hit rate (pinned)	60 ms	1100

That's a 13× TTFT improvement at the high end. Most user-perceived latency in a chat interface comes from TTFT, not from decode rate — the first tokens are what people wait for.

What hits don't give you

The cache doesn't help with:

• Single-request applications. If every user sends a unique prompt with no shared prefix, there's nothing to cache.
• Decode time. The cache only affects prefill. Long generations still take their decode time.
• Memory pressure on the model itself. The cache uses extra GPU memory; pushing it too large reduces room for everything else.

Bottom line

Most production LLM applications are leaving 50%+ of their potential cache hit rate on the table because of preventable prompt-construction mistakes. The fix is a one-day audit: identify the static prefix, canonicalize it, sort everything dynamic to the end, measure the hit rate under load. The ROI is immediate — both latency and throughput improve, and the GPU starts looking less saturated.