Lesson 25. Graceful Shutdown and Health

Mission link: A deploy restarts every instance. If shutdown drops in-flight requests, every deploy is a small outage, and this is the code that decides.
Primary source: http.Server.Shutdown
Prerequisites: Lesson 23, Lesson 19

Warm-up

1. ▢ What lets you raise log verbosity on a running process without restarting it?

2. ▢ Where does a long-lived worker learn that it should stop?

Know this

Shutdown has an order, and getting it wrong loses work:

1. Receive the signal. SIGTERM from the orchestrator, SIGINT from a terminal.
2. Fail the readiness probe so the load balancer stops sending new requests, and wait long enough for it to notice.
3. Stop accepting connections, while letting in-flight requests finish.
4. Wait for background workers to finish their current unit of work.
5. Close dependencies, meaning database, queues and a flushed logger, in reverse order of construction.
6. Give up after a deadline and exit anyway.

Step 6 is not optional. Something will eventually hang, and a process that refuses to die is worse than one that drops a request.

Signals

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()

signal.NotifyContext cancels the context on the first matching signal, which turns the whole of Lesson 18 into your shutdown mechanism: everything already taking this context already knows how to stop. A second signal is the operator saying they are done waiting: after stop(), the default behaviour returns and the next SIGINT kills the process.

Draining the server

srvErr := make(chan error, 1)
go func() { srvErr <- srv.ListenAndServe() }()

select {
case err := <-srvErr:
    return err                       // failed to start, or crashed
case <-ctx.Done():                   // signal received
}

shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
    return fmt.Errorf("shutdown: %w", err)
}

Shutdown closes listeners, waits for idle keep-alive connections to close, and waits for active requests to complete. It returns when everything is done or when its context expires, which is why that context must not be the cancelled one. Deriving the shutdown timeout from a context that is already cancelled aborts instantly and drops every in-flight request. It is the most common bug in this code.

ListenAndServe returns http.ErrServerClosed on a clean shutdown, so treat that value as success rather than as an error.

The shutdown timeout should be shorter than the orchestrator's grace period. Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds (30 by default), then SIGKILL. A 60-second shutdown timeout inside a 30-second grace period is a longer way of writing SIGKILL.

Workers and dependencies

Run the server and the workers under one errgroup so any of them failing brings the rest down:

g, gctx := errgroup.WithContext(ctx)
g.Go(func() error { return worker.Run(gctx) })
g.Go(func() error { return serve(gctx, srv) })
if err := g.Wait(); err != nil { ... }

Then close dependencies after g.Wait() returns, in reverse construction order, the database last, because a draining request may still need it. defer db.Close() in run gets this right for free, since defers unwind in reverse.

Liveness and readiness are different questions

Getting these backwards is a classic outage: a readiness check that pings the database, wired up as liveness, restarts every instance when the database has a brief hiccup, turning a recoverable dependency blip into a full outage with a cold cache.

Liveness should check almost nothing. Return 200 if the process can serve. Readiness may check dependencies the service cannot work without, and must return failure as soon as shutdown begins, before the listener closes.

mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)                  // liveness: am I alive
})
mux.HandleFunc("GET /readyz", func(w http.ResponseWriter, r *http.Request) {
    if shuttingDown.Load() || db.PingContext(r.Context()) != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
})

The gap between failing readiness and closing the listener matters. Load balancers notice on their own schedule, so sleeping a few seconds there, before calling Shutdown, is what actually prevents dropped requests during a deploy.

Practice

1. ▢ What is wrong with srv.Shutdown(ctx) where ctx is the one cancelled by the signal?

2. ▢ Why fail readiness before closing the listener, rather than at the same time?

3. ▢ Which check belongs in the liveness probe?

   • a) That the database responds to a ping
   • b) That the process can serve a response
   • c) That the message queue has capacity
   • d) That the cache has been warmed up

4. ▢ Your shutdown timeout is 60 seconds and Kubernetes' grace period is 30. What happens?

5. ▢ Interleaving Lesson 21: shutdown hangs until the timeout and the process is killed. Where do you look?

Real-world reps

• [ ] Build the full shutdown path: signal.NotifyContext, a goroutine running ListenAndServe, a fresh-context Shutdown. Start a slow request, press ctrl-c, and confirm the request completes.
• [ ] Break it deliberately by passing the cancelled context to Shutdown, and watch the same request get dropped. The difference is one argument.
• [ ] Tomorrow: check the shutdown timeout and the grace period for a service you operate. If nobody knows one of the two numbers, that is the finding.

Going further

• http.Server.Shutdown
• signal.NotifyContext
• errgroup
• Concurrency Patterns
• Resources

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.