Lesson 30. Benchmarks You Can Trust

Mission link: "Optimises from a profile and proves the win with benchstat" is the stage 5 exit criterion. This is the proving half, and most benchmarks people write do not prove anything.
Primary source: testing package: Benchmarks
Prerequisites: Lesson 28

Warm-up

1. ▢ Where does a fuzz failure get written, and why commit it?

2. ▢ What does t.Helper() change about a failure?

Know this

A benchmark lives beside the tests and runs on demand:

func BenchmarkParse(b *testing.B) {
    for b.Loop() {
        Parse("1h30m")
    }
}

go test -bench=BenchmarkParse -benchmem ./...

b.Loop() arrived in Go 1.24 and is now the form to write. It replaces for i := 0; i < b.N; i++, and it fixes two long-standing problems:

• Setup and cleanup run once per -count, not once per timing iteration, so expensive fixtures no longer need b.ResetTimer gymnastics.
• Parameters and results are kept alive, so the compiler cannot delete the call as dead code.

That second point is why so many old benchmarks report impossible numbers. With b.N and a result nobody uses, the optimiser is entitled to remove the work entirely and you measure an empty loop. The traditional defence was assigning to a package-level sink variable; b.Loop makes it unnecessary.

The output

BenchmarkParse-10        8123456           147.2 ns/op        32 B/op          1 allocs/op

-10 is GOMAXPROCS. Then iterations, nanoseconds per operation, and, with -benchmem, bytes and allocations per operation. Allocations per operation is usually the number that matters, because it is stable across machines and it is what you can actually act on. Nanoseconds move with the CPU, the thermal state and whatever else is running.

One run is not a measurement

A single number tells you nothing about variance. Run it repeatedly and compare with benchstat:

go test -bench=. -benchmem -count=10 ./... > old.txt
# make the change
go test -bench=. -benchmem -count=10 ./... > new.txt
benchstat old.txt new.txt

benchstat, from golang.org/x/perf/cmd/benchstat, reports the difference with a confidence interval and tells you when the change is statistically indistinguishable from noise. That verdict is the deliverable. "It went from 147ns to 141ns" is not a result; it is one sample from a noisy distribution.

Reduce the noise you control: close other applications, disable turbo or plug in the laptop, and never benchmark inside a container sharing a busy host. A quiet machine and -count=10 beats a clever methodology on a busy one.

Sub-benchmarks

b.Run mirrors t.Run, which is how you measure across input sizes:

for _, n := range []int{10, 1000, 100000} {
    b.Run(strconv.Itoa(n), func(b *testing.B) {
        data := makeData(n)          // once per -count, thanks to b.Loop
        for b.Loop() {
            Process(data)
        }
    })
}

Scaling behaviour is more informative than any single point. A function that is fine at 10 and quadratic at 100,000 is the bug you are looking for, and one benchmark size cannot show it.

What benchmarks cannot tell you

A microbenchmark measures one function on warm caches with no contention, no GC pressure from the rest of the program, and no network. Real gains and real regressions frequently live in the parts it excludes.

So the order is: profile the whole system to find where the time goes, benchmark the specific function to prove the fix. Reversing it produces a heavily optimised function that was never the problem, which is the most common way engineering time gets spent on nothing.

Practice

1. ▢ Why might for i := 0; i < b.N; i++ { Parse("3h") } report a suspiciously small number?

2. ▢ You changed a function and the benchmark went from 147 ns/op to 141 ns/op. What have you shown?

3. ▢ Which metric is most stable across different machines?

   • a) Nanoseconds per operation for the call
   • b) Total iterations the benchmark completed
   • c) Allocations per operation from -benchmem
   • d) Wall-clock seconds the benchmark ran

4. ▢ Why profile before benchmarking rather than the other way round?

5. ▢ Interleaving Lesson 19: you benchmark a RWMutex change and it looks slightly better. What is missing?

Real-world reps

• [ ] Write a benchmark with b.Loop, run it with -benchmem -count=10, and put the output through benchstat against itself. Seeing the noise floor of your own machine is worth doing once.
• [ ] Take a function that concatenates strings in a loop, benchmark it, switch to strings.Builder, and prove the win with benchstat. The allocation count will move more than the time.
• [ ] Tomorrow: install benchstat (go install golang.org/x/perf/cmd/benchstat@latest) if you have not, so it is there when you need it.

Going further

• testing package: Benchmarks
• benchstat
• Go 1.24 testing.B.Loop
• Toolchain Commands
• Resources

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.