LoRA from first principles: rank, alpha, and why low-rank works

You don't need to update every weight in a 70-billion-parameter model to teach it a new domain. You usually don't even need to update 1% of them. That's the bet behind LoRA — Low-Rank Adaptation — and it's been a remarkably durable bet.

The mechanic is simple enough to write on a napkin: freeze the base model. For each weight matrix W you want to adapt, allocate two new matrices A and B whose product BA has the same shape as W but is much smaller in total parameters. During the forward pass, use W + BA. During training, only A and B get gradients. Done.

This post answers the three questions we get asked on every engagement that involves LoRA: why does it work, how do you pick rank and alpha, and when does it stop being the right tool.

Why low rank works at all

When you fine-tune a base model on a specific domain, the change in weights is structured. It's not random noise; it's an update that pushes the model toward your data distribution. Empirically — and this is the load-bearing observation — that update lives in a low-dimensional subspace of the full weight space.

The 2021 LoRA paper (Hu et al.) showed that for a wide range of tasks, the effective rank of the fine-tuning update is small enough that you can decompose it as BA where A ∈ R^{r×k} and B ∈ R^{d×r} with r ≪ min(d, k). Total trainable parameters drop by a factor of roughly (d×k) / (r×(d+k)) — for a 4096×4096 attention projection at rank 8, that's a 256× reduction.

Why does the update live in a low-dimensional subspace? Two hand-wavy reasons:

1. The base model already encodes most of what your domain needs. A pre-trained 7B model has seen contracts, SQL, customer-support transcripts, all of it. Adaptation moves the emphasis, not the substance.
2. Your training data has limited "directions" it can push. A 10K-example SFT corpus carries maybe a few dozen meaningful axes of variation. Rank 8 has 8 axes — close enough.

Both of these break in specific cases (see "When LoRA stops working" below), but on most real engagements they hold.

Rank

Rank r is the single most important knob. Picking it badly will either underfit (rank too low) or waste compute and memory (rank too high without measurable gain).

We use this rough heuristic on engagements:

Task character	Recommended starting rank
Format shift only (output style, JSON schema)	4-8
Domain vocabulary + light reasoning shift	8-16
Heavy domain shift (legal, biomedical, code)	16-32
Multi-task or instruction-tuning over many tasks	32-64

These are starts. The real answer is to sweep three values (e.g. 8, 16, 32) on a held-out eval set and pick the elbow. Anything above 64 we treat as suspicious — you're probably better off doing a full fine-tune or grabbing a bigger base model.

A common mistake: assuming higher rank always helps. We've seen rank 64 underperform rank 16 on small datasets because the extra capacity overfits the training set noise.

Alpha

Alpha is the scaling factor: the actual update applied at inference is (α/r) × BA. The intent is to decouple the magnitude of the update from the rank, so that doubling rank doesn't double the effective update size.

The convention is α = 2r (so the scaling is 2), but the original paper used α = r (scaling 1), and people argue about both. Our pragmatic take:

• Start with α = 2r. It's the default in PEFT and most tutorials, and it works fine.
• If your loss is unstable (oscillating, spiking), drop alpha. Cut it in half.
• If your loss plateaus high, raise alpha (or raise the learning rate — they have similar effects).

Alpha is not a magic dial. If you're tuning it for more than a couple of sweeps you're probably masking a different problem (bad data, wrong learning rate, undertrained for the epoch count).

Target modules

The third decision: which weight matrices get a LoRA adapter? Options range from "only the query and value attention projections" (the LoRA paper's setup) to "every linear layer in the model."

Practical observations:

• Attention projections (q_proj, k_proj, v_proj, o_proj) are the highest leverage. Most papers and most engagements that work, work with these.
• Adding LoRA to MLP layers (gate_proj, up_proj, down_proj) helps for harder tasks. Quantization-aware fine-tunes (QLoRA) often target all linear layers because the base is so heavily quantized that you need adaptation everywhere.
• Skip the embedding layer. It rarely helps and complicates merging.

In PEFT this looks like:

from peft import LoraConfig

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

lora_dropout is worth setting to 0.05 if your dataset is small; it's a cheap regularizer. bias="none" means we don't train bias terms — they're tiny and rarely matter.

What this looks like at the matrix level

If you've internalized the napkin description, here's the slightly more concrete version. For an attention projection W ∈ R^{d×k}:

forward pass:     y = (W + (α/r) × B @ A) @ x
training pass:    only ∂L/∂A and ∂L/∂B propagate

A ∈ R^{r×k} is initialized with Gaussian noise; B ∈ R^{d×r} is initialized to zero. Because B = 0 at the start, BA = 0, which means the model behaves exactly like the base at step zero. Training then nudges B away from zero in whatever direction reduces loss.

This zero initialization on B is non-obvious but critical. If both were initialized randomly, the first forward pass would already deviate from the base, and the loss landscape would be much messier.

Merging

At inference time you have two choices:

1. Keep A and B separate and add BA to W at runtime. Slightly slower forward pass (one extra matrix multiply per adapted layer).
2. Merge once: compute W' = W + (α/r) × BA and ship W' as the new weight matrix. Same shape, same inference cost as the original.

Most production setups merge. The exception is adapter swapping — if you have multiple LoRAs trained for different tasks (one for SQL, one for customer support, one for code), you can keep them separate and load whichever you need per request. llama.cpp's --lora-scaled flag is exactly this pattern.

When LoRA stops working

A few honest failure modes we've hit:

Insufficient capacity for the task. If your domain is genuinely far from the base model's distribution (rare medical sub-specialty, a non-English language the base barely saw), rank 32 won't cut it. Either move to full fine-tuning or pick a base that's closer to your data.

Conflicting tasks in one adapter. Training a single LoRA on "answer SQL questions" + "write customer-support replies" produces an adapter that's mediocre at both. Either train two adapters and route between them at inference, or do multi-task training carefully with explicit task tokens.

Format collapse. The model learns the output format (JSON, specific phrasings) but loses general capability. Symptom: it answers questions about Paris correctly but the answer is now in a bizarre house style. Cause: training data was too narrow. Fix: mix in some general-instruction examples (1-5%) during fine-tuning to preserve the base's behavior on out-of-domain prompts.

Optimizer state explosion in QLoRA. This is the QLoRA paper's territory — when you quantize the base to 4-bit and use paged Adam, you can still OOM on a 24GB card if rank is too high or batch size is too big. We hit this on a 70B + rank 64 attempt; dropping to rank 32 and gradient accumulation of 16 fixed it.

What to measure

Loss going down is necessary but not sufficient. We always insist on:

• A domain-specific eval set. Held out, drawn from the same distribution as the training set but never seen during training. Score it once at the start (base model), once at the end (LoRA-merged).
• A general-capability eval set. MMLU, HellaSwag, or your own basket of out-of-domain prompts. Catches format collapse.
• At least one human-graded sample. Pick 20 prompts at random, eyeball the base vs. LoRA outputs side by side. Quantitative metrics miss subtle regressions.

If your domain eval went up but your general eval dropped more than 2-3 points, you're overfitting. Lower rank, more regularization, or mix in general data.

Where to start

A reasonable first attempt for any new engagement:

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

Train for 3 epochs at learning rate 2e-4, cosine schedule, batch size 4 with gradient accumulation of 4 (effective 16). Eval after each epoch on your domain test set. If you're still going down at epoch 3, try 5. If you're flat by epoch 2, train less and tune alpha or rank instead.

That's a reasonable place from which most engagements diverge into something more specific. The fact that this default works on so many tasks is itself a load-bearing data point about LoRA — most fine-tuning problems really do live in a low-rank subspace.