LLMLingua compression for a 4k-token system prompt

Most production system prompts are long. A typical enterprise deployment has 2,000-5,000 tokens of system prompt: persona instructions, output format examples, edge-case guidance, retrieved RAG context, few-shot examples. Every request pays the prefill cost on that prompt — and you pay the API token cost on it too.

LLMLingua compresses the prompt without losing the answer quality. This walkthrough takes a real-shaped 3,800-token system prompt and compresses it to ~850 tokens (4.5×) while keeping answer quality flat on a held-out QA set.

What it actually does

LLMLingua-2 (Microsoft Research, 2024) uses a small compression model (BERT-shaped, ~300M params) to identify low-information tokens in a long prompt and drop them. The compression model is a token classifier: for each input token, output the probability it's "necessary" for the downstream task. Keep tokens above a threshold, drop the rest.

Empirically the compression model learns that filler words, redundant phrasings, and verbose connectives are dropable, while content words, named entities, and structured elements are not. The result reads weirdly to a human but is fine for an LLM.

The setup

In Blueprint's Compress card:

• Input: the long system prompt we want to compress (a 3,800-token persona + RAG-context-style prompt)
• Target ratio: start at 4× (250 tokens / 1000 input tokens kept)
• Compression model: LLMLingua-2-bert-base-multilingual-cased-meetingbank (the default, public, MIT-licensed)
• Eval set: 100 question/expected-answer pairs we'll use to measure accuracy before/after

Hit Compress.

What runs under the hood

Blueprint's Compress card spawns the Python sidecar, which loads the compression model and runs:

from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
    use_llmlingua2=True,
)

compressed = compressor.compress_prompt(
    long_prompt,
    rate=0.25,           # keep 25% of tokens
    force_tokens=["\n", "?"],   # preserve structure markers
)

print(compressed["compressed_prompt"])

The compression runs in under a second on CPU. No GPU needed; this is a tiny model relative to the LLMs we're feeding it into.

What the compressed prompt looks like

Original (excerpt):

You are an expert legal assistant who has been carefully trained to help users with contract review questions. When you receive a question about a contract, you should carefully analyze the contract text and provide a detailed, accurate answer based on what is actually written in the contract. Make sure to cite specific sections when relevant...

After LLMLingua-2 at 4×:

expert legal assistant trained help users contract review questions. receive question contract carefully analyze contract text provide detailed accurate answer based actually written contract. cite specific sections relevant...

It reads like badly transcribed speech. It works because LLMs are remarkably good at reconstructing meaning from sparse keyword sequences — the model has seen enough examples to fill in the missing connectives.

Measuring quality

The compression card runs both the original and compressed prompts against our held-out QA set and scores each response against the reference answer.

Results on a 100-question domain-specific contract QA set:

At 4×, we lose 2pp of accuracy and save 3,000 tokens per request. That's the sweet spot for this prompt. At 6× the quality starts to drop noticeably. At 10× it falls off a cliff — the prompt is no longer carrying enough signal.

The right ratio is application-specific. For lightly-constrained generation (creative writing, summarization) you can often push 8-10× without much loss. For tightly-constrained generation (code, structured outputs) 3-4× is the realistic ceiling.

Cost impact

On an OpenAI deployment, prompt tokens are billed at ~$0.15-2.50/M depending on model. Saving 3,000 tokens per request at 100,000 requests/month = 300M tokens saved = $45 (mini) to $750 (4o) per month. Multiply by 10× requests / month for serious traffic.

On a self-hosted deployment, the cost isn't dollars-per-token — it's compute time. 3,000 fewer prefill tokens per request × 100K requests = 3M tokens of prefill saved per month. At 600 tokens/sec server-wide that's 5,000 fewer compute seconds, ~80 minutes of GPU time per month freed up for more requests.

The bigger effect for self-hosted is TTFT. Cutting prompt tokens by 4× cuts prefill time by ~4× (linear in token count). User-perceived latency drops accordingly.

Where it doesn't work

A few cases where LLMLingua is the wrong tool:

Structured prompts. JSON, XML, exact format templates. Compressing these breaks the structure the model expects. Force-keep the structural tokens via the force_tokens parameter, or skip compression.

Multi-lingual where one language compresses worse than another. The compression model is trained mostly on English. Non-English text compresses less efficiently and may lose quality faster.

Very short prompts. Below ~500 tokens, the compression overhead (running the BERT model) is comparable to the savings on the LLM side. Not worth it.

Prompts where every word matters. Code generation prompts where the variable names and types are load-bearing. Compression that drops a variable name produces wrong output.

Production setup

Compression runs once per stable prefix, not once per request. The system prompt is static; compress it during deployment, store the compressed version, serve that. The compression model isn't on the hot path.

# Once, at deploy time:
compressed_sys_prompt = compressor.compress_prompt(SYSTEM_PROMPT, rate=0.25)["compressed_prompt"]
cache.set("compressed_system_prompt", compressed_sys_prompt)

# Per request:
prompt = compressed_sys_prompt + user_query
response = llm.generate(prompt)

For dynamic portions (retrieved RAG context per query), you can compress those too — the model loads in less than 1s on CPU, so compressing 1K-token retrieved chunks per request adds negligible latency to overall response time.

What to try next

• Compare LLMLingua-2 against the simpler LongLLMLingua for long-context tasks
• Cascade with prompt cache: the compressed system prompt is now small enough that cache hit rate matters less, but cache + compress together is still strictly better
• Compare per-request vs. cached compression latency to decide which to use

The pattern: compress the boring parts (instructions, examples, generic context) aggressively; keep the precious parts (the actual question, structured data) intact.