Introduction

When a Go service receives SIGTERM (from Kubernetes, systemd, or docker stop), it has a limited window (usually 30 seconds) to gracefully shut down. Without proper signal handling, the process exits immediately, abandoning in-flight requests, dropping database connections without cleanup, and leaving temporary resources behind. The context package and signal.NotifyContext provide a clean pattern for propagating shutdown signals throughout the application, but many services miss critical cleanup steps like draining HTTP servers, flushing log buffers, and waiting for background goroutines to complete.

Symptoms

``` # Kubernetes pod logs show: # No shutdown logs at all - process killed immediately # Client receives: 502 Bad Gateway

# Or partial shutdown: 2026/04/09 10:00:00 Received SIGTERM, shutting down... # But database connections not closed # Background goroutines still running ```

Or resource leaks after restart:

bash
# Database shows stale connections from previous process
SELECT count(*) FROM pg_stat_activity WHERE state = 'idle';
-- Count keeps growing with each restart

Common Causes

  • No signal handler: Process exits immediately on SIGTERM
  • HTTP server not shut down gracefully: Server.Shutdown() not called
  • In-flight requests aborted: Connections dropped without response
  • Background goroutines not tracked: No waitgroup for worker goroutines
  • Database connections not closed: Connection pool not released
  • Shutdown timeout too short: Cleanup takes longer than Kubernetes grace period

Step-by-Step Fix

Step 1: Set up signal handling with context

```go package main

import ( "context" "log" "net/http" "os" "os/signal" "syscall" "time" )

func main() { // Create context that cancels on SIGTERM or SIGINT ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT) defer stop()

server := &http.Server{ Addr: ":8080", Handler: setupRoutes(), }

// Start server in background go func() { log.Println("Server starting on :8080") if err := server.ListenAndServe(); err != http.ErrServerClosed { log.Fatalf("Server error: %v", err) } }()

// Wait for shutdown signal <-ctx.Done() log.Println("Shutdown signal received")

// Create shutdown context with timeout shutdownCtx, cancel := context.WithTimeout(context.Background(), 25*time.Second) defer cancel()

// Graceful shutdown if err := server.Shutdown(shutdownCtx); err != nil { log.Printf("Server shutdown error: %v", err) } log.Println("Server stopped gracefully") } ```

Step 2: Track and drain background workers

```go import "sync"

type WorkerPool struct { wg sync.WaitGroup }

func (wp *WorkerPool) Start(ctx context.Context, workers int) { for i := 0; i < workers; i++ { wp.wg.Add(1) go func(id int) { defer wp.wg.Done() for { select { case <-ctx.Done(): log.Printf("Worker %d shutting down", id) return default: // Do work processJob() } } }(i) } }

func (wp *WorkerPool) Stop() { wp.wg.Wait() log.Println("All workers stopped") }

// Usage in main workers := &WorkerPool{} workers.Start(ctx, 4)

// After server shutdown workers.Stop() ```

Step 3: Clean up all resources in order

```go func gracefulShutdown(server *http.Server, db *sql.DB, workers *WorkerPool) { // 1. Stop accepting new requests (server.Shutdown) // 2. Wait for in-flight requests to complete // 3. Stop background workers // 4. Close database connections // 5. Flush log buffers

shutdownCtx, cancel := context.WithTimeout(context.Background(), 25*time.Second) defer cancel()

server.Shutdown(shutdownCtx) log.Println("HTTP server shut down")

workers.Stop() log.Println("Workers stopped")

if err := db.Close(); err != nil { log.Printf("Error closing database: %v", err) } log.Println("Database connections closed") } ```

Prevention

  • Always use signal.NotifyContext for signal handling in Go 1.16+
  • Set Kubernetes terminationGracePeriodSeconds to at least 30 seconds
  • Use server.Shutdown() instead of server.Close() to drain requests
  • Add a readiness probe that returns 503 during shutdown to stop traffic
  • Track all goroutines with WaitGroup to ensure clean exit
  • Close database connection pools to release server-side resources
  • Test graceful shutdown locally with kill -TERM <pid> before deploying