Lesson 9. Rank, Alpha and Initialisation

Mission link: The mission asks you to justify every hyperparameter from the task. These three are where most people repeat folklore instead.
Primary source: Docs: LoRA, Hugging Face PEFT
Prerequisites: Lesson 8

Warm-up

1. ▢ Write the LoRA forward equation.

2. ▢ Adapter parameter count for one targeted layer?

3. ▢ Why does a merged adapter cost nothing at inference?

Know this

Initialisation, and why it is not arbitrary

A is initialised randomly, either Gaussian or Kaiming-uniform. B is initialised to exactly zero.

Therefore BA = 0 at step zero, so ΔW = 0, so the adapted model is bit-for-bit the base model before the first update. This is a deliberate and important property:

• Training starts from a known-good model rather than a perturbed one.
• There is no initial shock to pretrained behaviour, so no early damage to undo.
• Any degradation you observe is something training did, not something initialisation did.

A's entries are visible and none of them matter: every one of them is multiplied by a zero on its way into the product.

The asymmetry is necessary. If both were zero, the gradient of both would be zero and nothing would ever learn: the product needs one factor non-zero to carry signal. If both were random, ΔW would start as structured noise injected into every targeted layer.

This is also why LoRA tolerates much higher learning rates than full fine-tuning. You are not nudging weights that already encode everything the model knows; you are growing a term from zero.

Alpha and the scaling factor

α (lora_alpha) is a fixed scalar. The update is scaled by α/r.

The point of dividing by r was to make the effective update magnitude roughly independent of rank, so that changing rank would not force you to re-tune the learning rate. What matters is the ratio α/r, not either number alone.

That last row is a trap worth naming: raising rank while leaving α fixed reduces the scale of the update. People raise rank hoping for more expressive power, forget α, and conclude that higher rank hurts. It did not; they quietly turned the adapter down by 8×.

The conventional starting point is α = 2r. Treat it as a convention with a rationale, not a discovery.

rsLoRA (arXiv:2312.03732) argues the α/r divisor over-damps at high rank and proposes α/√r instead, which keeps learning stable as rank grows. If you plan to work at rank 64 or above, this is the setting to know:

LoraConfig(r=64, lora_alpha=16, use_rslora=True, ...)

Choosing rank

Rank is capacity. It bounds how complex an update the adapter can express.

Two diagnostics that beat guessing:

Underfitting. Training loss plateaus above where you need it and more epochs do not help. Not enough capacity: raise rank, or add target modules (usually the better move, per Lesson 10).

Overfitting. Training loss keeps falling while held-out loss rises. Too much capacity for the data, or too many epochs. Lower rank, add dropout, get more data, or stop earlier.

Note the asymmetry with the memory lesson: rank costs very little memory (Lesson 7), so the cost of extra rank is overfitting risk and a little compute, not fitting in memory. That inverts the instinct many people bring from the era when memory was the binding constraint.

Dropout

lora_dropout applies dropout to the adapter input. Typical values are 0.0 to 0.1. Reach for it when held-out loss is rising and you cannot get more data. On large datasets it is often unnecessary and simply slows convergence.

The config

config := {
    rank: 16,
    alpha: 32,               # α/r = 2
    dropout: 0.05,
    adapt_bias: false,
    target_modules: [...],   # Lesson 10
}

model := attach_adapter(model, config)
print(model.trainable_parameter_count())

That last line is not optional. It is the check that your target modules matched anything at all, and it should agree with the arithmetic you did in Lesson 8. Read the parameter names from the installed version's documentation rather than from memory, because this surface changes between releases.

Practice

1. ▢ Why is B initialised to zero and A randomly, rather than both to zero or both randomly?

2. ▢ You raise rank from 8 to 64 and leave α at 16. Loss barely moves. Diagnose.

3. ▢ Training loss flattens at a level too high to be useful, and a fourth epoch does not help. What is this, and what are your two options?

4. ▢ Which pairing gives the strongest update scale?

   • a) r = 8 with alpha = 8
   • b) r = 8 with alpha = 32
   • c) r = 32 with alpha = 32
   • d) r = 64 with alpha = 64

5. ▢ Why can LoRA use a learning rate ten to a hundred times higher than full fine-tuning?

6. ▢ When would you not increase rank even though the adapter is underfitting?

Real-world reps

• [ ] Build three configs, (8, 16), (16, 32) and (64, 16), and write down the α/r for each and what you predict will happen. Keep the predictions.
• [ ] Print print_trainable_parameters() for each and reconcile against your Lesson 8 arithmetic.
• [ ] Tomorrow: find a published fine-tuning config in the wild, work out its α/r, and decide whether the author chose it or inherited it.

Going further

• Docs: LoRA, Hugging Face PEFT
• Paper: "A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA", Kalajdzievski, arXiv:2312.03732
• Blog: "Practical Tips for Finetuning LLMs Using LoRA", Sebastian Raschka
• LoRA hyperparameters

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.