Lesson 21. Goroutine Leaks

Mission link: This is the failure that closes stage 3: finding a leak in code you did not write, and naming the guarantee that was missed. It is also the most common Go production bug.
Primary source: Never start a goroutine without knowing how it will stop, Dave Cheney
Prerequisites: Lesson 20

Warm-up

1. ▢ What are the three conditions for a data race?

2. ▢ go test -race passes. What does that prove?

Know this

A goroutine leak is a goroutine that will never finish, because it is blocked on something that will never happen. The runtime cannot collect it, so its stack stays, and so does every object its stack references. A leak that grows with traffic is a slow memory leak with a scheduler cost attached.

Unlike a deadlock, nothing reports it. all goroutines are asleep fires only when every goroutine is blocked, and a leaking service has a perfectly healthy HTTP listener.

The four shapes

Send with no receiver. The most common one, and it hides behind an early return:

func search(ctx context.Context, queries []string) (string, error) {
    ch := make(chan string)          // unbuffered
    for _, q := range queries {
        go func() { ch <- lookup(q) }()   // blocks until someone receives
    }
    return <-ch, nil                 // takes the first, abandons the rest
}

The first result returns. Every other goroutine is parked on its send forever. Fix by giving the channel capacity len(queries) so every send completes, or by cancelling the rest with a context.

Receive with no sender. A goroutine ranging a channel nobody closes, or waiting on a result the producer failed to send because it returned an error first.

No cancellation case. A worker selecting only on its job channel: when shutdown comes, nothing tells it to stop, so the process hangs until the shutdown timeout kills it.

A forgotten Ticker. time.NewTicker without defer ticker.Stop() while you still hold the reference. Since Go 1.23 an unreferenced timer is collected, but a ticker held in a struct field is referenced, and the goroutine reading it never ends.

Finding them

The goroutine profile is the first tool, and it works on a running service:

curl 'http://localhost:6060/debug/pprof/goroutine?debug=2' > goroutines.txt

debug=2 prints every goroutine's full stack with how long it has been blocked. A leak looks like ten thousand goroutines parked on the same line of your code. Take two profiles ten minutes apart and diff the counts: a leak grows, normal load does not.

Since Go 1.27 there is a purpose-built profile: goroutineleak, in runtime/pprof and at /debug/pprof/goroutineleak. The runtime uses the garbage collector to find goroutines blocked on a primitive that is unreachable from any runnable goroutine, which cannot possibly be unblocked, and is therefore leaked rather than merely slow. It was an experiment in Go 1.26 and is generally available in 1.27. It does not find every leak: a channel still reachable through a global, or through a running goroutine's locals, looks live to the collector.

In tests, go.uber.org/goleak fails a test that finishes with goroutines it did not start. One line in TestMain covers a whole package.

Not leaking, by construction

• Every goroutine gets a ctx, and every long-lived loop has a case <-ctx.Done().
• Every send has a receiver that is guaranteed to arrive, or a buffer big enough that it cannot block, or a select with ctx.Done().
• Whoever starts a goroutine is responsible for its shutdown. errgroup makes that ownership explicit; a bare go f() in the middle of a function usually does not.
• Close response bodies. defer resp.Body.Close() is a connection leak rather than a goroutine leak, and it shows up in the same incident.

The question to ask in review is the one in the title of the primary source: how does this goroutine stop? If the answer takes more than a sentence, it probably does not.

Practice

1. ▢ Find the leak.

   ch := make(chan string)
   for _, q := range queries {
       go func() { ch <- lookup(q) }()
   }
   return <-ch
   

2. ▢ Why does the runtime not report this, when it reports all goroutines are asleep - deadlock!?

3. ▢ Which command shows you goroutines blocked in your service right now?

   • a) go test -race ./... -count 1
   • b) curl localhost:6060/debug/pprof/goroutine?debug=2
   • c) go build -gcflags=-m ./... 2>&1
   • d) GODEBUG=gctrace=1 ./svc 2>&1 | tail

4. ▢ A worker selects only on its job channel. Shutdown hangs for 30 seconds and then the process is killed. Explain the sequence.

5. ▢ Interleaving Lesson 18: a request times out after 2 seconds. Three goroutines it started are still running 30 seconds later. Is this a leak?

Real-world reps

• [ ] Write the leaking search function, print runtime.NumGoroutine() before and after, and watch the number stay high. Then fix it with a buffer and watch it drop.
• [ ] Add net/http/pprof to a scratch service, leak a hundred goroutines deliberately, and read ?debug=2. Learning to read that output when nothing is on fire is the point.
• [ ] Tomorrow: add goleak.VerifyTestMain(m) to one package you own and run its tests. Whatever it reports was already there.

Going further

• Never start a goroutine without knowing how it will stop, Dave Cheney
• Go 1.27 goroutine leak profile
• net/http/pprof
• go.uber.org/goleak
• Concurrency Patterns
• Resources

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.