Lesson 14. Quantisation from First Principles

Mission link: QLoRA is LoRA plus one idea. This lesson is that idea, taught on its own so the next one is short.
Primary source: Paper: "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale", Dettmers et al., arXiv:2208.07339
Prerequisites: Lesson 5, quantisation

Warm-up

1. ▢ Bytes per parameter in fp32, bf16 and 4-bit?

2. ▢ For a 7B base with a 20M adapter, where does the training memory go?

3. ▢ Why does merging into a quantized base lose quality?

Know this

Quantisation stores numbers using fewer bits than they were trained in. Recall from the glossary that in this workspace it always refers to the frozen base, not the adapter.

The mechanism

To store a high-precision tensor in a small integer type, you need a mapping between the real range and the integer range. Two schemes:

Symmetric, absmax. Take the largest absolute value, scale so it lands at the integer maximum.

s = absmax(x) / 127          # for int8
q = round(x / s)             # store q, an int8
x̂ = q · s                    # recover, approximately

Asymmetric, affine. Also store a zero-point offset, so the range need not be centred on zero.

s = (max(x) − min(x)) / 255
z = round(−min(x) / s)
q = round(x / s) + z

Weight distributions in transformers are roughly symmetric and centred near zero, so symmetric absmax is the common choice for weights.

Either way, the error is rounding error: each value moves to the nearest representable level. Sixteen levels for 4-bit, 256 for 8-bit.

The outlier problem

Here is what makes naive quantisation fail on transformers. A single very large value in a tensor sets absmax, which sets the scale, which spreads all the representable levels out to cover a range that almost every other value never uses. The typical values then all round into a handful of levels, and most of the tensor's information is destroyed by one outlier.

This is not hypothetical. Transformer activations develop systematic large-magnitude features, and this observation is the core of the LLM.int8() work: a small number of outlier dimensions carry disproportionate importance, and handling them naively destroys quality at scale.

Blockwise quantisation, the fix

Do not quantise a whole tensor with one scale. Split it into small contiguous blocks of 64, 128 or 256 elements, and give each block its own scale.

per-tensor:  one absmax  → one outlier ruins everything
per-block:   one absmax per 64 → one outlier ruins 64 values

Both rows have sixteen levels, spaced identically. The only thing that changed is how much numeric range those sixteen levels are asked to cover, and that is set by the largest value in whatever the scale is computed over. Spend the grid on a range the data never visits and the data arrives at a handful of levels; spend it on the range the data actually occupies and each value keeps a level of its own.

The damage is contained. This is why blockwise quantisation is standard and per-tensor quantisation is not.

It has a cost, and this cost is the whole subject of the next lesson. Each block needs its scale stored, and that scale is a real number:

4-bit weights, block size 64, fp32 scales:
  weights = 0.5 bytes/param
  scales  = 4 bytes per 64 params = 0.0625 bytes/param
  total   = 0.5625 bytes/param, i.e. 4.5 bits, not 4

Twelve percent overhead on top of a 4-bit budget, purely for bookkeeping. Hold that number.

Storage precision versus compute precision

A quantized weight is not used in its quantized form. The forward pass dequantises a block back to bf16, does the matmul, and discards the dequantized copy.

stored:   4-bit + per-block scale
computed: bf16

So quantisation buys memory, and costs compute: a dequantisation step per block, per forward pass. Whether the net effect is faster or slower depends entirely on whether you were memory-bound or compute-bound, and on how well the dequantisation kernel is optimised for the hardware you are on.

That last point is where hardware genuinely enters. The arithmetic is identical everywhere. What differs by backend is whether a fused, optimised kernel exists for a given bit-width and block size, or whether the library falls back to a slower generic path. Lesson 16 gives the current picture.

What quantisation is for, here

Two uses that must not be confused:

Quantisation for inference shrinks a model so it fits and serves cheaply. The weights are quantized and that is the artifact you ship. This is out of scope for this workspace as a topic in its own right.

Quantisation for training, which is what QLoRA does, shrinks the frozen base so that adapter training fits. The adapter stays at higher precision, and the artifact you ship may be merged into a full-precision base afterwards. The base's precision is a training-time budget decision, not a property of the result.

Practice

1. ▢ Quantise the values [0.1, −0.3, 0.2, 8.0] to int8 with symmetric absmax. What happens to the first three?

2. ▢ Why does blockwise quantisation help, and what does it cost?

3. ▢ Total bytes per parameter for 4-bit weights, block size 128, fp32 scales?

4. ▢ In a quantized forward pass, which precision is the matmul done in?

5. ▢ Why is transformer quantisation harder than it looks from the arithmetic alone?

6. ▢ Distinguish quantisation for inference from quantisation for training.

Real-world reps

• [ ] Implement absmax int8 quantisation and dequantisation for a tensor by hand in a few lines. Measure the maximum and mean error.
• [ ] Insert one large outlier and measure the error again. Then redo it blockwise with block size 64 and compare.
• [ ] Tomorrow: compute the true bytes-per-parameter for 4-bit with block sizes 32, 64, 128 and 256, and note where the overhead becomes negligible.

Going further

• Paper: "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale", Dettmers et al., arXiv:2208.07339
• Paper: "8-bit Optimizers via Block-wise Quantization", Dettmers et al., arXiv:2110.02861
• Lesson 15. NF4 and Double Quantisation

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.