Prompt cache in production: what hits, what misses, and how to design around it
Most production LLM traffic shares a 1-4 kilo-token system prompt across every request. A prompt cache turns that shared prefix into a one-time cost — but only if your application sends the prefix the same way every time.