Prompt cache

What it does

Every LLM request goes through two phases:

1. Prefill — the model reads your entire prompt and computes a KV (key-value) tensor for every token. This is where most of the wall-clock time goes on long prompts.
2. Decode — the model generates output tokens one at a time, reusing the KV state from step 1.

The prompt cache stores the KV tensor for prompt prefixes you've seen before. When a new request arrives whose prompt starts with a prefix already in the cache, the engine skips most of the prefill work and goes straight to decode. Time-to-first-token on a cache hit drops by 80–95% for prompts where the cached prefix is the bulk of the input.

When it helps

Production traffic almost always has a shared prefix nobody notices:

• A 1–4 kilo-token system prompt repeated on every request
• A retrieval block appended to every query in a RAG pipeline (when the retrieved docs are the same — common for FAQ-style traffic)
• Few-shot examples at the start of every prompt in a classification pipeline

In each case the cache hit rate approaches 100% and the cost of the prefix essentially disappears.

It does not help when:

• Every request has a unique prompt (one-shot creative generation, conversational use where the message history changes each turn)
• The prefix changes frequently (live system-prompt A/B testing, dynamic retrieval where every result set is fresh)

How to use it

1. Open Dashboard → Inference.
2. In the Prompt cache card, enable the cache by toggling the switch.
3. Set the prefix length — the longest prefix the cache should track. Match this to your actual shared-prefix size: if your system prompt is 1,500 tokens, set it to 2,000; setting it too small (e.g. 500) means a 1,500-token system prompt won't hit the cache at all.
4. Send a request through the Inference tab to seed the cache. The second identical request should show a TTFT drop in the metrics card.

Common pitfalls

• Prefix-length too small is the #1 cause of "the cache isn't helping me." The engine only caches up to the configured length; if your real prefix is longer, you get zero hits.
• Whitespace and tokenizer differences matter. A trailing newline or an extra space can change the tokenization of the boundary and miss the cache. Normalize your prompt construction.
• Cache eviction: the cache has a finite size. If you serve many distinct prefixes, the LRU policy kicks the older ones out. Check the cache-stats card for an eviction count; if it's high relative to hits, raise the cache size or reduce the number of distinct prefixes.

Measuring the impact

The Dashboard's Performance card shows TTFT P50 and P95. The fastest way to confirm the cache is working: turn it off, run 10 requests, note the P50 TTFT. Turn it on, send the same 10 requests twice (the first run seeds the cache). The second run's P50 should be 5–20× lower on requests whose prefix is fully cached.