Efficient LLM Post-Training: LoRA, QLoRA & PEFT

Chapter 1: Alignment: SFT & RLHF

Before we get to efficient fine-tuning, we need to understand why fine-tuning happens at all. A pretrained language model predicts the next token on internet text. It does not follow instructions, stay safe, or give helpful answers by default. Post-training is the process of aligning the model's behavior with human intent.

Supervised Fine-Tuning (SFT) is the simplest form: collect (prompt, ideal response) pairs, then train the model with the standard next-token prediction loss on those responses only. The model learns "when given this kind of prompt, produce this kind of output." Llama-2's SFT dataset includes both helpfulness and safety demonstration pairs.

SFT works, but it only covers what's in your demonstration dataset. It doesn't capture what humans actually prefer among multiple valid responses. That's the gap RLHF (Reinforcement Learning from Human Feedback) fills. Humans rank pairs of model responses, a reward model learns these preferences, then the language model is fine-tuned to maximize the reward — with a KL penalty to prevent it from "hacking" the reward model by drifting far from the pretrained distribution.

In all three cases (SFT, RLHF, DPO), the underlying operation is updating the model weights via gradient descent. And that's where the fine-tuning wall from Chapter 0 bites us. SFT on a 7B model still requires 84 GB at full precision. RLHF is worse — you're running three models simultaneously. The efficiency techniques in the rest of this lesson apply to all these post-training regimes.

Chapter 2: Early PEFT Methods

The PEFT research community attacked the fine-tuning wall from several directions before LoRA became dominant. Understanding these methods clarifies why LoRA won — each earlier approach had a critical weakness.

BitFit (Zaken et al., 2021) is the simplest idea: freeze all weights, train only the bias terms. BERT-base has 110M parameters, but only 0.1M biases — that's 1,000× fewer trainable parameters. For small to medium datasets, BitFit is surprisingly competitive with full fine-tuning. For larger datasets it falls behind. Reason: bias terms have limited capacity. They shift activations but can't rotate the representation space.

Adapter (Houlsby et al., 2019) inserts small bottleneck modules inside each Transformer layer — after the multi-head attention projection and after each feed-forward layer. Each adapter: down-project from d to r (r ≪ d), apply non-linearity, up-project back to d, add skip connection. Only adapter weights are trained; all original weights are frozen. For 1000 tasks with a 7B model: full fine-tuning = 1000 × 14 GB = 14 PB storage; adapters = 1000 × ~14 MB = 14 GB. A million-fold reduction.

Prompt Tuning / Prefix Tuning (Lester 2021, Li & Liang 2021) prepend learned "soft prompt" tokens to the input instead of changing weights. Prompt tuning adds learnable tokens to the first layer only. Prefix tuning adds them to every layer's key/value matrices. Both are elegant but share a problem: they consume context window. If you prepend 100 learned tokens to a model with a 2048-token window, you lose ~5% of usable context, and inference is slower because the attention spans those extra tokens.

All early PEFT methods share a training efficiency win — fewer trainable params, smaller optimizer states. But they add inference overhead. The open question entering 2022 was: can we build a PEFT method that adds no inference overhead at all?

Chapter 3: LoRA: Low-Rank Algebra

LoRA (Hu et al., ICLR 2022) solves the inference-latency problem with one algebraic trick: the fine-tuning update is low-rank, and low-rank matrices can be added to the original weight before inference begins — no new forward-pass overhead at all.

Here is the core idea. A pretrained weight matrix W has shape d×k (for a key/query/value projection in a Transformer, d = k = 4096 in LLaMA-2-7B). When you fine-tune, you update W ← W + ΔW. Full fine-tuning computes and stores the entire d×k update ΔW — all 4096 × 4096 = 16.8 million numbers.

LoRA hypothesizes that ΔW has low intrinsic rank: the fine-tuning update can be captured by a matrix whose rank r is much smaller than d. Formally, instead of learning ΔW directly, LoRA learns two small matrices:

The forward pass becomes:

Initialization: A is initialized with random Gaussian values; B is initialized to zeros. This ensures ΔW = B·A = 0 at the start of training — the model behaves identically to the pretrained base at initialization, so fine-tuning starts from a stable baseline.

The scaling factor α/r (where α is a hyperparameter, often set to the rank r or 2r) controls the learning rate of the LoRA update relative to the frozen weights. A larger α makes the adapter update more aggressive.

Chapter 4: LoRA: Parameter Math

Let's work out the exact numbers. A single linear layer in LLaMA-2-7B (e.g., the Q-projection) has d = 4096, k = 4096. Full fine-tuning trains d × k = 16,777,216 parameters for this layer — and stores that many gradient values and 2× that many Adam states in FP32.

LoRA at rank r = 8 trains: A is r × k = 8 × 4096 = 32,768 values; B is d × r = 4096 × 8 = 32,768 values; total = 65,536 parameters. That's a 256× reduction for this layer alone.

In a full LLaMA-2-7B model with 32 layers, LoRA is typically applied to the Q, K, V, and O projection matrices (4 per layer × 32 layers = 128 matrices). Total LoRA parameters at r = 8: 128 × 65,536 = 8,388,608 ≈ 8.4M — less than 0.12% of the 7B total. The entire adapter fits in 16 MB (FP16).

python
import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    """Frozen W with trainable low-rank update B @ A."""
    def __init__(self, d_in, d_out, rank=8, alpha=16):
        super().__init__()
        # Frozen pretrained weight — not updated
        self.W = nn.Parameter(torch.randn(d_out, d_in), requires_grad=False)
        # Trainable LoRA adapter
        self.A = nn.Parameter(torch.empty(rank, d_in))   # r x k
        self.B = nn.Parameter(torch.zeros(d_out, rank))  # d x r — zeros!
        nn.init.kaiming_uniform_(self.A, a=5**0.5)        # Gaussian-like init for A
        self.scale = alpha / rank                         # α/r scaling factor

    def forward(self, x):
        # x @ W.T + (x @ A.T) @ B.T * scale
        # At init: B=0, so output = x @ W.T exactly
        base = x @ self.W.T
        lora = x @ self.A.T @ self.B.T * self.scale
        return base + lora

    def merge(self):
        """Fold adapter into W — zero inference overhead."""
        with torch.no_grad():
            self.W += self.scale * (self.B @ self.A)
        self.A = self.B = None   # free adapter memory

Chapter 5: QLoRA: 4-bit Fine-Tuning

LoRA dramatically reduces the trainable parameters — but the frozen base model still occupies 14 GB in FP16. On a 24 GB RTX 4090, that leaves only 10 GB for gradients, optimizer states, activations, and the LoRA adapters. For a 7B model it's tight. For a 65B model it's completely impossible.

QLoRA (Dettmers et al., NeurIPS 2023) combines LoRA with aggressive quantization of the frozen base. The key insight: the frozen weights don't need to be in FP16 during training. They're never updated — they're just read to compute forward-pass activations. If you store them in 4-bit, the 7B base occupies only 3.5 GB instead of 14 GB.

The QLoRA training loop works like this:

Three innovations make QLoRA work:

1. NF4 quantization — a new 4-bit data type optimized for normally-distributed weights (explained in Ch.6)
2. Double quantization — quantize the quantization constants themselves, saving ~0.37 bits/param more
3. Paged optimizers — NVIDIA's unified memory manages optimizer state spikes by swapping to CPU RAM when GPU memory overflows, preventing OOM crashes during long-context samples

Result: A 65B model (normally requires 780 GB for full fine-tuning) can be fine-tuned on a single 48 GB A40 GPU with QLoRA. The resulting model (Guanaco) achieves 99.3% of ChatGPT's performance on the Vicuna benchmark.

Chapter 6: NF4 & Double Quantization

Standard INT4 assigns equally-spaced values between -8 and 7. But LLM weight values are not uniformly distributed — they follow a bell-shaped (approximately normal) distribution, concentrated near zero. If you use INT4's uniform grid, most of your quantization levels cluster where the weights aren't, wasting representational budget.

NormalFloat (NF4) constructs its 16 levels by placing them at the quantiles of the standard normal distribution — so each level represents an equal fraction of the probability mass. More levels near zero (where most weights live), fewer levels at the extremes. Same 4 bits, lower average quantization error.

The exact NF4 values (16 levels):

Notice they're denser near zero and sparser at the extremes — exactly matching where normal-distributed weights actually live. Before quantizing a weight tensor, it's scaled to [-1, 1] using the absolute maximum, then each value is mapped to its nearest NF4 level.

Double Quantization (DQ): NF4 uses one FP32 scaling constant per group of 64 weights (e.g. 32-bit float / 64 weights = 0.5 bits/param overhead). Double quantization quantizes these scale constants to 8-bit floats with a second-level 256-group constant. This reduces the overhead from 0.5 bits/param to ~0.127 bits/param — saving an additional 0.37 bits per parameter, worth ~3 GB on a 65B model.

python
# NF4 quantization (simplified implementation)
import numpy as np

NF4_LEVELS = np.array([
    -1.0, -0.6962, -0.5251, -0.3949, -0.2844, -0.1848, -0.0911, 0.0,
     0.0796,  0.1609,  0.2461,  0.3379,  0.4407,  0.5626,  0.7230, 1.0
])  # quantile positions of N(0,1)

def quantize_nf4(w, group_size=64):
    """Quantize weight tensor w to NF4 in groups of 64."""
    w_flat = w.reshape(-1, group_size)
    abs_max = np.abs(w_flat).max(axis=1, keepdims=True)  # [N/64, 1]
    w_norm = w_flat / (abs_max + 1e-8)                     # scale to [-1, 1]
    # Find nearest NF4 level for each value
    indices = np.argmin(np.abs(w_norm[:,:,None] - NF4_LEVELS[None,None,:]), axis=2)
    return indices.astype(np.uint8), abs_max.astype(np.float32)  # 4-bit + FP32 scales

def dequantize_nf4(indices, abs_max):
    """Reconstruct BF16 weights from NF4 for a matmul."""
    w_norm = NF4_LEVELS[indices]                # map indices back to float levels
    return (w_norm * abs_max).reshape(-1)     # rescale and flatten

Chapter 7: Memory Budget Lab

Let's compare the three training regimes side by side with exact numbers. For LLaMA-2-7B (7 × 10⁹ params) fine-tuning a single epoch on your task dataset:

For LLaMA-2-65B (65 × 10⁹ params):

Chapter 8: Multi-Adapter Serving & BitDelta

You fine-tuned 1000 specialized variants of your 7B model with LoRA — one per customer, enterprise, or domain. At deployment time, you have 1000 sets of adapters. The base model is shared, immutable, and loaded once in GPU memory. When a request arrives, you swap in the appropriate adapter, run the forward pass, and return the result.

This is multi-adapter serving: one frozen base on one GPU, many swappable LoRA adapters (16 MB each). Instead of 1000 × 14 GB = 14 TB of model storage and 1000 GPUs, you need 14 GB (one base) + 1000 × 16 MB = 16 GB total — a 875× storage reduction.

BitDelta (Liu et al., NeurIPS 2024) takes this idea further: can we compress the fine-tuning delta (ΔW = W_fine − W_pretrained) even more aggressively than LoRA? The intuition is that fine-tuning adds less information than pretraining — the delta should be more compressible. BitDelta compresses ΔW to 1 bit per weight with a per-tensor scaling factor. The binary representation (±1 scaled by γ) is fine-tuned jointly with γ.

Storage: a 7B model fine-tune with BitDelta requires only 7B bits = 875 MB — vs 14 GB in FP16. For multi-tenant serving, this means the per-task delta is 16× smaller than LoRA adapters. The "the more you serve, the more you save" effect: a fused binary GEMM kernel multiplies the 1-bit delta with the batch, and since the kernel is more memory-bandwidth efficient than FP16 matmul, throughput increases as batch size grows.

Chapter 9: Connections & PEFT Cheat Sheet

You've now seen the complete PEFT landscape. Here is how the methods compare when you actually need to choose one.

Rank Selection Guide

There is no single right rank. The empirical guidance from the LoRA paper and subsequent ablations:

• r = 4–8: Sufficient for most NLP classification and generation tasks. Use when data is abundant (the model generalizes its adaptation) or when storage is constrained.
• r = 16–32: Better for complex reasoning tasks, code generation, or domain shifts requiring richer adaptation. Common default in practice.
• r = 64+: Diminishing returns beyond r=64 for most tasks. Approximates full fine-tuning at high enough rank. Use for extreme domain shift.
• Which matrices: Apply to Q, K, V, O in attention. Can also add FFN up/down projections. More matrices + low r often beats fewer matrices + high r.

Related Lessons in This Series

"Full fine-tuning says: update everything. LoRA says: the update is a small perturbation in a low-dimensional subspace — parameterize only that. QLoRA says: compress even the frozen base while we're at it. BitDelta says: once fine-tuned, the delta itself compresses to a single bit per weight. Each step is another way of saying the same thing: adaptation is cheap information." — MIT 6.5940 course notes