Lesson 17. Defending a Latency Budget End to End

Mission link: This is the mission's final lesson: quoting and defending a p99 latency budget end to end means diagnosing a miss by walking back through cache, batching, and quantization in the right order, and defending the fix with the numbers each of those lessons taught how to measure.
Primary source: Docs: vLLM Documentation, vLLM Project
Prerequisites: Lesson 16, Lesson 9, Lesson 2, KV cache

Warm-up

1. ▢ Write lesson 2's capacity ceiling formula from memory.

2. ▢ What three numbers does lesson 9 say a quantization choice needs to be defended with?

Know this

Diagnosis follows the phase, then the lever

A missed p99 budget is a symptom, not a diagnosis. The first split is which phase is missing budget: p99 TTFT points at prefill and scheduling (lesson 5's head-of-line blocking, chunked prefill's chunk size), while p99 inter-token latency points at decode (lesson 6's batch size, lesson 3's cache precision, lessons 7 to 9's weight quantization). Checking the wrong phase's levers wastes effort: tuning the chunk size does nothing for a decode-bound ITL problem, and shrinking the batch does nothing for a TTFT problem caused by a large prefill stalling the queue.

Once the phase is identified, the lever order follows cost: the cheapest fix that doesn't cost accuracy comes first (batch size, since lesson 6 already showed how to compute the largest batch a budget allows), then a cache-precision change (lesson 3's fp8 KV cache, which costs no weight accuracy), then weight quantization (lessons 7 to 9, which does cost measured accuracy and should only be reached for once the cheaper levers are exhausted or insufficient).

A worked chain

A server serving a 13B model on one 80 GB GPU measures, from a realistic 2,000-request benchmark (lesson 16), a p99 TTFT comfortably under budget and a p99 inter-token latency of 80 ms against a 50 ms budget. TTFT being fine rules out lesson 5's scheduling problems; this is a decode-phase, ITL problem. The team checks the batch size first: it's already at the figure lesson 6 would defend for this workload, so shrinking it further would cost throughput the workload needs. Next, they quantize the weights to int8 (lesson 7), which lowers ITL to 65 ms, measuring the accuracy delta against the unquantized model as lesson 9 requires. Still over budget, they weigh two remaining options: drop the batch size further (free of accuracy cost, but costs throughput) or quantize to int4 with AWQ (frees more decode time, at a further, measured accuracy cost). Lesson 9's framework decides it: whichever option closes the remaining gap without paying for more than the budget actually needs. If a smaller batch size still meets the workload's required concurrency, that is the cheaper fix; only if it doesn't does the further accuracy cost of int4 get paid.

What the final defense has to cite

A configuration that now meets its p99 budget is not yet defended until it names three things: the measured p99 figures themselves, from a sample large and realistic enough to be trustworthy (lesson 16); the memory and batch-size reasoning behind whatever capacity ceiling and batch size the configuration settled on (lessons 2 and 6); and the accuracy number measured for whatever quantization, if any, was applied to get there (lesson 9). A configuration that hits its latency target by luck, with none of these three written down, is exactly as undefended as one that misses it.

Practice

1. ▢ A server's p99 inter-token latency is 80 ms against a 50 ms budget, while p99 TTFT is comfortably under budget. Name the first two things worth checking, in order, and why in that order.

2. ▢ After quantizing to int8, p99 ITL drops to 65 ms, still above the 50 ms budget. The team is deciding between shrinking the batch size further or quantizing to int4. What does lesson 9's framework say should decide between them?

3. ▢ The team settles on int8 quantization with a batch size that keeps concurrency where the workload needs it, achieving p99 ITL of 48 ms and p99 TTFT of 210 ms, both measured from 2,000 requests with a prompt-length distribution matching production traffic. What three things does their final defense need to cite?

4. ▢ Which claim is true of diagnosing a missed p99 latency budget end to end?

   • a) Any flag can be changed at random until the measured number improves
   • b) The diagnostic order follows which phase is missing budget, then which lever that phase's mechanics point to, cheapest first
   • c) A single benchmark run is sufficient evidence to defend a configuration change
   • d) Quantization should always be the first thing tried, regardless of which phase is missing budget

Real-world reps

• [ ] For a workload you have in mind, or a real one you're serving, walk this lesson's diagnostic order end to end: identify which phase (if either) is missing budget, check the cheapest lever first, and write down the three things your final defense would need to cite.
• [ ] If you've been building a real deployment across this workspace's lessons, gather its actual measured p99 TTFT and p99 ITL, from a benchmark sized and shaped the way lesson 16 describes, and check them against a latency budget you'd defend.
• [ ] Tomorrow: revisit this workspace's mission in README.md and confirm, in your own words, that you can now do what it describes: stand up a serving stack and defend the latency budget it produces.

Going further

• Docs: vLLM Documentation, vLLM Project
• Article: "Transformer Inference Arithmetic", Kipply
• Resources

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.