QLoRA on a consumer GPU: fitting a 13B fine-tune in 24GB

The first time we ran a 13B LoRA on a single 4090 we expected it to OOM. It did, three times — but the fourth attempt fit comfortably with ~6GB of headroom. QLoRA isn't magic; it's three techniques stacked together that each cut a different part of the memory bill.

This is the recipe we've shipped on five client engagements over the past six months. It works on a 4090 (24GB), an A6000 (48GB), or anything with at least 24GB of VRAM. For 70B fine-tunes you want 48GB+ and a willingness to wait.

The three techniques

QLoRA's original paper combines:

1. 4-bit NormalFloat (NF4) quantization of the frozen base model
2. Paged optimizer state — pages out unused Adam moments to CPU RAM
3. Double quantization — quantize the quantization constants themselves

Plus the standard tricks: LoRA for parameter-efficient adapters, gradient checkpointing to trade compute for memory, mixed-precision training (bfloat16 forward, float32 accumulator).

Together, these turn a 26GB model (13B × 2 bytes for bf16) into something that fits — with optimizer state, gradients, activations, and a batch of training examples — in 24GB.

The recipe

import torch
from transformers import (
    AutoModelForCausalLM, AutoTokenizer,
    BitsAndBytesConfig, TrainingArguments,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

MODEL_ID = "meta-llama/Llama-3.3-13B"

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
model = prepare_model_for_kbit_training(model)

lora = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora)

training_args = TrainingArguments(
    output_dir="./out",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,           # effective batch = 16
    gradient_checkpointing=True,             # trade compute for memory
    optim="paged_adamw_8bit",                # paged + 8bit optimizer
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,                                # mixed precision
    logging_steps=10,
    save_strategy="epoch",
    eval_strategy="epoch",
)

trainer = SFTTrainer(
    model=model,
    train_dataset=train_ds,
    eval_dataset=eval_ds,
    args=training_args,
    tokenizer=tokenizer,
    max_seq_length=2048,
    dataset_text_field="text",
)

trainer.train()
trainer.model.save_pretrained("./out/adapter")

That fits on a 4090. The relevant settings, in order of memory impact:

Setting-by-setting impact

load_in_4bit=True + bnb_4bit_quant_type="nf4"

This is the headline. A 13B model in bfloat16 is 26GB. In NF4 it's 7GB. Saved: ~19GB. NF4 is information-theoretically optimal for normally-distributed weights and outperforms fp4 and int4 consistently — there's no reason to pick anything else.

bnb_4bit_use_double_quant=True

The quantization constants themselves (one per group of weights) take space. Double quantization quantizes them too. Saves another ~0.4GB on a 13B model. Free win.

optim="paged_adamw_8bit"

Adam stores two moments (m, v) per trainable parameter. For LoRA's small parameter set this is small, but paged_adamw_8bit adds two protections:

• 8-bit storage of moments (4× smaller than float32)
• CPU paging when GPU memory pressure spikes (CUDA Unified Memory under the hood)

Without paging, you'll see OOMs mid-training on a 4090. With it, you trade a bit of speed for survival.

gradient_checkpointing=True

Activations during the forward pass dominate memory for long sequences. Gradient checkpointing recomputes them during the backward pass instead of storing them — roughly halves activation memory, costs ~30% extra compute. On a 4090 with max_seq_length=2048, this is the difference between fitting and not fitting.

per_device_train_batch_size=2 + gradient_accumulation_steps=8

Effective batch is 16, but only 2 sequences live in VRAM at any moment. The other 7 batches accumulate into the same gradients before the optimizer step. Mathematically equivalent to batch 16, memory equivalent to batch 2.

max_seq_length=2048

Each token of context costs O(seq_length × hidden_dim) of activation memory. Doubling sequence length roughly doubles the per-step memory. For most domains 2048 is enough; if you need 4096 either drop batch size or use a 48GB card.

What it looks like running

On a 4090 training a 13B Llama on a 10K-example SFT dataset:

3 hours is a reasonable evening run. 70B at the same recipe on an A6000 (48GB) takes about 8 hours — overnight territory.

Failure modes and fixes

OOM in the first 100 steps. Almost always activation memory. Either drop max_seq_length to 1024 or enable gradient_checkpointing if you somehow turned it off.

OOM during eval but not training. Eval runs without gradient checkpointing by default. Either pass eval_gradient_checkpointing=True (newer TRL versions) or eval on a 4090 with a smaller per_device_eval_batch_size=1.

Loss going to NaN. Usually bnb_4bit_compute_dtype mismatch. Force it to bfloat16 on Ampere+ cards. On older cards (T4, V100) drop to float16 but expect more instability.

Adapter saving but loading wrong shape. Make sure the inference path also uses BitsAndBytesConfig to load the base in 4-bit before applying the adapter, OR merges the adapter into a bf16 base. Mixing 4-bit base + bf16 adapter at inference produces broken outputs.

What you can't do on 24GB

• 70B fine-tunes at any reasonable rank. Even with all tricks, 70B in 4-bit is 35GB before training state.
• Long-context fine-tuning (>4K). The activation memory cliff is steep.
• Multi-GPU training is awkward — bitsandbytes' paging interacts badly with PyTorch DDP. Stick to single-GPU.

For those: rent an H100 for $2-5/hr, do the run in a few hours, terminate. Blueprint's Phase C will eventually provision these on demand from the GUI; today you're SSH-ing into Runpod.

What this unlocks

Once a 4090 can do a 13B QLoRA in an evening, the gating constraint on fine-tuning shifts from compute to data quality. Most engagement time is now spent curating the SFT corpus, not waiting on training. That's the right place for the bottleneck to be — it forces the engagement to ask "is this dataset actually representative" instead of "can we afford to train."