Lesson 10. Choosing Target Modules

Mission link: Target modules matter more than rank, and the received wisdom on them is out of date. This is a place where you can be measurably better than the average practitioner.
Primary source: Docs: "LoRA Without Regret", Hugging Face TRL
Prerequisites: Lesson 2, Lesson 9

Warm-up

1. ▢ What does α/r control, and what does r alone control?

2. ▢ Which sublayer holds most of a transformer block's parameters?

3. ▢ What does zero-initialised B guarantee?

Know this

What the original paper did, and why that is not advice

The LoRA paper adapted only the attention projections, in its experiments mostly W_q and W_v, and left the MLP alone. That choice propagated into years of tutorials, and it is where target_modules=["q_proj", "v_proj"] comes from.

Two things to understand about it. It was a reasonable finding under a memory budget and on the tasks measured. And it is not a general result about where adaptation should happen. Treating a 2021 ablation as a default is the single most common inherited mistake in this field.

The case for the MLP

From Lesson 2: the MLP holds roughly four fifths of each block's parameters. There is a substantial body of interpretability work treating MLP layers as where much of a model's learned knowledge and feature detection lives, while attention routes information between positions. If your task needs the model to know or classify something differently, the parameters that encode that are disproportionately in the MLP, and an attention-only adapter cannot reach them.

The current default

Recent work converges on a blunt recommendation: adapt all linear layers, and give the adapter enough rank. The TRL write-up of the "LoRA without regret" result is explicit: applying LoRA to every linear layer, including the MLP, with sufficient rank, closes the gap to full fine-tuning on supervised fine-tuning workloads, while attention-only configurations do not.

# The modern default: let the library find every linear layer.
config := {rank: 64, alpha: 128, target_modules: "all-linear"}

# The explicit equivalent, for a Llama-style architecture.
config := {
    rank: 64,
    alpha: 128,
    target_modules: [
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
}

"all-linear" is the safer of the two, because it cannot go stale against an architecture whose module names differ from the list you memorised. Its cost is that you give up precise control and should check what it actually matched.

Why this is affordable now: from Lesson 7, adapter parameters are a negligible share of training memory. Restricting target modules to save memory was a real trade in 2021 and is close to a non-trade today. What you are actually trading is overfitting risk on small datasets.

The decision

Two modules people ask about

Embeddings and the output head. Usually left alone. Adapting them is expensive relative to benefit, and it is the one place where changing weights can shift the meaning of every token. The genuine exception is adding new special tokens, which requires touching the embedding table and is a different operation from adapting it.

Bias terms. bias="none" is the standard setting. Training biases as well ( "all" or "lora_only") is cheap but rarely the difference between a working and a failing run, and it complicates merging.

The check that catches the real error

Print what matched, every time:

model := attach_adapter(base_model, config)
print(model.trainable_parameter_count())
print(first 8 parameter names containing "lora")

A target-module name that matched nothing produces either an error or, depending on version, a model with no adapter at all, which trains, logs a loss, and learns nothing. A trainable-parameter count that disagrees with your Lesson 8 arithmetic is the earliest possible warning.

Practice

1. ▢ Why did the original paper target attention only, and why is that not a default today?

2. ▢ Your task requires the model to distinguish between domain concepts it currently conflates. Attention-only or all-linear?

3. ▢ You have 400 training examples. Argue for restricting target modules.

4. ▢ Which change is most likely to raise quality on a real task with a reasonable dataset?

   • a) Raising the adapter rank from 16 to 64
   • b) Extending targets to all linear layers
   • c) Adding LoRA dropout at 0.05 to the run
   • d) Training for one additional full epoch

5. ▢ You set target_modules=["c_attn"] and the run trains with loss barely moving. What do you check first?

6. ▢ Why is "all-linear" safer than an explicit module list?

Real-world reps

• [ ] Build an attention-only and an all-linear adapter at the same rank on the same model. Record both trainable-parameter counts and the ratio.
• [ ] Verify which modules "all-linear" matched by printing adapter parameter names. Note anything unexpected.
• [ ] Tomorrow: find three fine-tuning configs published online and note which target attention only. Judge whether the author chose that or inherited it.

Going further

• Docs: "LoRA Without Regret", Hugging Face TRL
• Blog: "Practical Tips for Finetuning LLMs Using LoRA", Sebastian Raschka: the target-module ablations
• LoRA hyperparameters
• Lesson 11. Your First Adapter

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.