LoRA training

What a LoRA is

LoRA (Low-Rank Adaptation) fine-tunes a small "delta" matrix on top of a frozen base model. Instead of updating all 7 billion weights of a 7B model, you train two low-rank matrices (typically rank 16-64) that approximate the weight update. The result:

• Adapter file is tiny — 50 MB for a typical 7B LoRA vs the 14 GB full-weight checkpoint
• Training is cheap — QLoRA (quantized base + LoRA adapter) trains a 7B on a single 24 GB consumer GPU; full fine-tune of the same model wants 80+ GB
• Adapters compose — multiple LoRAs trained on different domains can be loaded into the same base model

The cost: LoRA is less expressive than a full fine-tune. For most domain-adaptation work (your support transcripts, your SQL prompts, your contracts) the gap doesn't matter; for pre-training-scale distribution shifts, you'd want a full fine-tune.

When LoRA pays off

• Prompt engineering has plateaued. You've tweaked the system prompt 30 times and can't squeeze more accuracy. The model needs to LEARN your domain, not just be told about it.
• You have ≥1,000 labeled examples of (input, expected-output). Fewer than that and you're better off with better prompts.
• The base model is open enough to fine-tune. Llama 3, Qwen, Mistral, Phi — yes. Some commercial-license models — check first.

How to run a training job

1. Dashboard → Training card → New job.
2. Pick the base model (must be pulled) and the training data. Blueprint expects JSONL with {"input": "...", "output": "..."} pairs.
3. Set hyperparameters. Defaults are good for most domain-adaptation:
   • LoRA rank: 16 (raise to 32-64 for more capacity, more VRAM)
   • Alpha: 32 (typically 2 * rank)
   • Learning rate: 2e-4
   • Epochs: 3
   • Batch size: 1-4 (depends on GPU and sequence length)
4. Start. The job runs in the Python sidecar. Watch tokens/sec, loss, eval-loss in the live log.
5. On completion, the adapter file lands at ~/.blueprint/loras/<job-name>.gguf and shows up in the LoRA adapters list.

Training time scales with dataset size + epochs + base model size. Rough numbers for 1,000 examples:

• 7B base, 3 epochs, RTX 4090: ~45 min
• 7B base, 3 epochs, RTX 4060 Laptop: ~3 hours
• 70B base, 3 epochs, A100 80GB: ~2 hours (QLoRA)

Using the trained adapter

When configuring a serve, pick the LoRA adapter from the dropdown under Optional flags. llama-server loads the base model + applies the adapter at inference time. Use --lora-scaled (set in the config) to control adapter strength (1.0 = full effect, 0 = base only).

Common pitfalls

• Tiny dataset (< 100 examples): LoRAs need data. Sub-100 just learns to memorize the prompts; the model produces those exact outputs and nothing else.
• Mismatched chat template: training a chat-format LoRA on raw-text data (no role markers) produces an adapter that breaks the base model's chat format. Match your training data to how the base was trained.
• Overfitting: high training loss isn't the goal; low eval loss is. Watch the eval-loss curve — if it bottoms out and turns up while training loss keeps dropping, stop early.
• Adapter quality looks great in training, fails in production: the eval data was too similar to training data. Pull eval data from a different time window than training data.