Lesson 3. Tokenizers and Chat Templates

Mission link: The most common cause of a fine-tune that trains cleanly and behaves badly is a template mismatch, not a hyperparameter.
Primary source: Docs: Chat Templates, Hugging Face Transformers
Prerequisites: Lesson 1, Lesson 2

Warm-up

1. ▢ Which holds more parameters in a modern block, attention or the MLP?

2. ▢ How do you find a model's real module names?

3. ▢ What does cross-entropy loss measure?

Know this

The model does not see text. It sees integers. The tokenizer is the mapping between them, and it is part of the model: a checkpoint's tokenizer is not interchangeable with another's, even within the same family.

What a tokenizer is

Modern models use subword tokenization, usually byte-pair encoding. Training the tokenizer means starting from bytes and repeatedly merging the most frequent adjacent pair, producing a vocabulary where common words are single tokens and rare words split into pieces.

Three properties bite in practice:

Tokens are not words. the (with a leading space) and the are different tokens. Capitalisation changes tokenization. A number like 1234 may be one token or four, and this is why models are bad at arithmetic in ways that look arbitrary.

Token count is not character count. English averages roughly four characters per token; code, non-Latin scripts and unusual formatting are far less efficient. Your sequence-length budget in Lesson 7 is measured in tokens, so you must measure, not estimate.

Special tokens carry structure. Beginning-of-sequence, end-of-sequence, padding, and the role markers below are ordinary vocabulary entries that the model has learned to treat as structural. Emitting them by hand incorrectly is not a syntax error; it is a silent distribution shift.

Chat templates

A chat model was trained on conversations rendered into one flat token sequence by a specific, exact string format. For example, a model might have been trained on something shaped like:

<|im_start|>system
You are helpful.<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi.<|im_end|>

Another family uses entirely different markers. The format is not a convention you can approximate: it is the distribution the weights were fit to. Get a marker wrong, add a stray newline, or omit the system block when the model expects one, and the model is now operating slightly off-distribution. It will still produce fluent text, which is exactly why this failure is hard to spot.

The template ships with the tokenizer, as a Jinja string. Use it rather than writing the format yourself:

tokenizer := load_tokenizer("<model>")
messages := [
    {role: "system", content: "You are helpful."},
    {role: "user", content: "Hello"},
]

# For inference: leaves the sequence ready for the model to continue.
prompt := tokenizer.render(messages, add_generation_prompt: true)

# For training: the assistant turn is already present, so no generation prompt.
full := tokenizer.render(messages + [{role: "assistant", content: "Hi."}])

print(exact(prompt))
print(exact(full))

Printing the exact string, not a summarised view, matters. The difference between a correct and a broken template is often one whitespace character, and an ordinary print can hide it — in Python that means repr, not print, on the string.

The one rule

Train on the same rendering you will serve. If you fine-tune on your own hand-rolled format and then serve through the model's official template, you have trained one distribution and deployed another. The loss curve will look fine. The model will be subtly worse, and you will spend a day blaming rank.

add_generation_prompt is where this most often goes wrong: it belongs in inference, where you want the model to continue as the assistant, and not in a training example, where the assistant text is part of the sequence being learned.

Both renderings are the same pieces in the same places up to the assistant marker. The one that differs is the piece after it, which is exactly what the model is supposed to produce at inference and exactly what it is supposed to learn during training.

Base models have no template

If a base model's tokenizer has no chat template, that is not a bug, because there is no conversation format to preserve. You are choosing the format, so choose one, write it down, and use it identically in training and serving.

Practice

1. ▢ Why can't you swap a tokenizer between two checkpoints of the same model family?

2. ▢ You are building a training example. Should add_generation_prompt be True or False, and why?

3. ▢ Your fine-tune trains to a low loss but at serving time the model rambles past the end of its answer instead of stopping. Name the most likely cause.

4. ▢ A prompt is 700 English words. Roughly how many tokens, and would you trust that estimate for a training budget?

5. ▢ Which of these is safe to change between training and serving?

   • a) The exact chat template string used
   • b) The tokenizer shipped with the model
   • c) The sampling temperature used at generation
   • d) The special token marking a turn end

Real-world reps

• [ ] Take one model and render a template with your library's real chat-template call (e.g. apply_chat_template(..., tokenize=False) in Transformers), printing the exact string with whitespace included (e.g. Python's repr). Copy it into your notes.
• [ ] Tokenize the same paragraph in two different model families and compare token counts. Then tokenize a block of JSON and compare again.
• [ ] Tomorrow: deliberately break a template by dropping the end-of-turn marker, then prompt the model. Watch what it does. That is the failure you are learning to recognise.

Going further

• Docs: Chat Templates, Hugging Face Transformers
• Code: minbpe, Andrej Karpathy: byte-pair encoding in readable form
• Failure modes: template mismatch appears there
• Lesson 21. Building the Dataset

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.