Lesson 13. GGUF

Mission link: This is the workspace's turn to CPU and edge: GGUF is the format llama.cpp actually serves from, and understanding what it bundles and how it loads is the ground floor for everything stage 5 covers.
Primary source: Repo: llama.cpp, ggml-org
Prerequisites: Lesson 12, KV cache

Warm-up

1. ▢ Why does quantizing a model's weights speed up decode specifically?

2. ▢ What is the key difference between how GPTQ and AWQ (lesson 7) decide which weights to protect from quantization error?

Know this

One file instead of several

A model served through the Hugging Face ecosystem, the shape vLLM expects, usually arrives as several files: the weights themselves, a config.json describing the architecture, a tokenizer file, sometimes more. GGUF bundles all of that, weights, tokenizer, and the metadata needed to reconstruct the architecture, into a single binary file. On a GPU server with the full Python and Hugging Face stack already present, that separation is not a burden. On an edge device, where the goal is often to ship one file and run it with a small, dependency-light binary, a single self-contained file is exactly what the deployment needs.

GGUF has its own quantization family

GPTQ and AWQ (lesson 7) are GPU-oriented schemes, tuned for how GPU kernels dequantize and multiply. GGUF instead carries llama.cpp's own family of quantization types, named things like Q4_K_M or Q5_K_M. Reading the name: the leading number is the bit width (Q4 is roughly 4 bits per weight); K marks a k-quant method, which mixes precision within a tensor rather than quantizing every weight uniformly, spending more bits on the parts of the tensor found to matter more; the trailing letter (S, M, or L, small, medium, large) selects among a few preset configurations of that mixing, trading size against quality the same way lesson 8 described int8 versus int4 trading off, just within llama.cpp's own scheme rather than GPTQ's or AWQ's.

Memory-mapping instead of loading

Because GGUF lays weights out in a format the OS can read directly, llama.cpp can memory-map the file (mmap) rather than reading it into RAM and deserializing it the way a Python checkpoint loader does. Memory-mapping means the file's pages are only actually read from disk when something touches them, and the operating system's own page cache manages what stays resident, rather than the process claiming all of it upfront. On a GPU server backed by fast, plentiful system RAM, this matters less. On CPU and edge hardware, where RAM is often the tightest resource, avoiding an eager, full-file load, and letting the OS decide what to keep resident, is a real difference in how large a model a given device can even attempt to run.

Practice

1. ▢ Why does bundling weights, tokenizer, and architecture metadata into one GGUF file matter more for edge deployment than for a GPU server already running the full Hugging Face stack?

2. ▢ Decode the quantization name Q4_K_M: what does each part mean?

3. ▢ What does memory-mapping a GGUF file let llama.cpp avoid doing at startup, and why does that matter more on CPU/edge hardware than on a GPU server?

4. ▢ Which claim is true of GGUF's k-quant family compared to GPTQ or AWQ?

   • a) It is the same scheme as GPTQ, just renamed for llama.cpp
   • b) It is llama.cpp's own quantization family, distinct from the GPU-oriented GPTQ and AWQ schemes
   • c) It only supports 4-bit quantization, unlike GPTQ and AWQ
   • d) It requires a GPU to apply, the same as GPTQ and AWQ

Real-world reps

• [ ] Find a GGUF checkpoint of a model you know on a model hub, and read its file listing: note that it's a single file, or a small number of split parts, rather than the multi-file layout an HF checkpoint uses.
• [ ] Look at the available quantization variants for that same checkpoint (often several Q*_K_* options) and note the file size difference between two of them.
• [ ] Tomorrow: read one paragraph of llama.cpp's docs on mmap and note what flag, if any, controls whether it's used.

Going further

• Repo: llama.cpp, ggml-org
• Docs: "Optimizing inference", Hugging Face Transformers
• Resources

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.