VRAM math for any open LLM, from 7B to 405B
Three numbers and one ratio tell you whether a model fits on a given GPU. Here's the formula, with worked examples for Llama-3.3, Qwen 2.5, and DeepSeek across every common quant level.
Blog
Deep-dives on LLM optimization
Self-hosted vs API: the actual math at 1M / 10M / 100M tokens/month
API pricing is per-token and scales linearly. On-prem cost is mostly fixed. The crossover depends on usage, but it's lower than most people think. Here's the math at three usage tiers, with assumptions disclosed.
RAG vs fine-tuning vs both: a decision tree
RAG injects retrieved knowledge at inference time. Fine-tuning bakes it into the weights. Most production systems need some of each, and picking the wrong tool wastes engagement budget. Here's how we triage.
QLoRA on a consumer GPU: fitting a 13B fine-tune in 24GB
You don't need an H100 to fine-tune a 13B model. With 4-bit base quantization, paged optimizer state, and gradient checkpointing, a 4090 will train one in an evening. Here's the recipe and the trap doors.
Prompt cache in production: what hits, what misses, and how to design around it
Most production LLM traffic shares a 1-4 kilo-token system prompt across every request. A prompt cache turns that shared prefix into a one-time cost — but only if your application sends the prefix the same way every time.
Picking the right open model for your workload
There are eight open models worth considering in 2026. Picking the right one for your workload is mostly a function of three constraints: hardware, latency, and domain. Here's the decision matrix we use on engagements.
LoRA from first principles: rank, alpha, and why low-rank works
LoRA freezes the base model and learns two small matrices whose product gets added to the original weights. That's the whole idea. Here's why it works, how to pick rank and alpha, and the failure modes we've hit on real engagements.
When llama.cpp beats vLLM (and vice versa)
Both can serve open LLMs. They have very different strengths, and picking the wrong one for your workload costs you both throughput and money. Here's the decision tree we use on engagements.
What imatrix actually does (and why off-the-shelf Q4 leaves accuracy on the table)
Pre-quantized GGUFs from HuggingFace are calibrated against general web text. If your workload is anything else, you're paying an accuracy tax. imatrix lets you re-calibrate against your actual prompts — here's what changes and what to measure.
Building an evaluation harness that survives prod
An eval set isn't 200 hand-picked test cases. It's a continuously-updated representation of what your users actually send, scored by metrics that correlate with what they actually want. Here's the harness shape we ship.
Building a calibration corpus from your client's prompts
A custom imatrix is only as good as the calibration data behind it. Here's how we assemble a representative 2000-prompt corpus from a client's production logs without breaking PII rules or overfitting to the test set.
Break-even analysis: when on-prem GPUs beat reserved cloud
Cloud GPU rental at $1.50-3/hr looks expensive on paper. Owning an H100 looks expensive on the invoice. The crossover depends on utilization more than anything else — here's the math, with caveats.
Subscribe
Get new deep-dives in your inbox
Roughly one article a fortnight. Technical, no marketing fluff, unsubscribe any time. We never share your address.