Introduction

Go applications in Docker containers need to handle SIGTERM for graceful shutdown during deployments and scaling events. However, signals may not reach the Go process due to PID 1 semantics, shell wrappers, or incorrect signal notification setup. When signals are not caught, Docker waits for the shutdown timeout (default 10s) and then sends SIGKILL, abruptly terminating in-flight requests.

Symptoms

  • Docker logs show SIGKILL instead of graceful shutdown
  • In-flight requests are dropped during deployment
  • context deadline exceeded errors during rolling updates
  • Database connections not properly closed
  • Shutdown hooks never execute
bash
# Docker stops container
$ docker stop myapp
# After 10s timeout:
# Container killed with SIGKILL - no graceful shutdown occurred

Common Causes

  • Go process running as PID 1 in container (special signal handling)
  • Shell script wrapper (ENTRYPOINT shell form) eating signals
  • signal.Notify not called before signals arrive
  • Using wrong signal constants (syscall.SIGTERM vs os.Interrupt)
  • Signal channel buffer too small, dropping signals

Step-by-Step Fix

  1. 1.Set up proper signal handling:
  2. 2.```go
  3. 3.func main() {
  4. 4.ctx, stop := signal.NotifyContext(context.Background(),
  5. 5.syscall.SIGINT, syscall.SIGTERM)
  6. 6.defer stop()

// Start your server srv := startServer()

// Wait for signal <-ctx.Done() log.Println("Received shutdown signal")

// Graceful shutdown with timeout shutdownCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second) defer cancel()

if err := srv.Shutdown(shutdownCtx); err != nil { log.Fatalf("Server forced to shutdown: %v", err) } log.Println("Server exited gracefully") } ```

  1. 1.Fix Docker ENTRYPOINT to forward signals:
  2. 2.```dockerfile
  3. 3.# WRONG - shell form does NOT forward signals
  4. 4.ENTRYPOINT ./start.sh

# CORRECT - exec form replaces shell with the process ENTRYPOINT ["./myapp"]

# Or if you need a shell script, use exec: # start.sh #!/bin/sh exec "$@" # Replaces shell with the actual command ```

  1. 1.Use tini or dumb-init as PID 1 init system:
  2. 2.```dockerfile
  3. 3.FROM golang:1.22 AS builder
  4. 4.# ... build steps ...

FROM debian:bookworm-slim RUN apt-get update && apt-get install -y tini COPY --from=builder /app/myapp /myapp

# tini properly forwards signals to child process ENTRYPOINT ["/usr/bin/tini", "--"] CMD ["/myapp"] ```

  1. 1.Handle multiple signal channels correctly:
  2. 2.```go
  3. 3.// WRONG - signal sent to first registered channel only
  4. 4.sigChan1 := make(chan os.Signal, 1)
  5. 5.sigChan2 := make(chan os.Signal, 1)
  6. 6.signal.Notify(sigChan1, syscall.SIGTERM) // This one gets the signal
  7. 7.signal.Notify(sigChan2, syscall.SIGTERM) // This one does NOT

// CORRECT - use a single channel with buffer sigChan := make(chan os.Signal, 1) signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)

go func() { sig := <-sigChan log.Printf("Received signal: %v", sig) }() ```

  1. 1.Implement graceful HTTP server shutdown:
  2. 2.```go
  3. 3.srv := &http.Server{
  4. 4.Addr: ":8080",
  5. 5.Handler: mux,
  6. 6.}

go func() { if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed { log.Fatalf("Server failed: %v", err) } }()

// Wait for signal <-ctx.Done()

// Graceful shutdown - stops accepting new connections, waits for active shutdownCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second) defer cancel()

if err := srv.Shutdown(shutdownCtx); err != nil { log.Printf("Server shutdown error: %v", err) } ```

Prevention

  • Always use exec form ["binary"] for ENTRYPOINT/CMD in Dockerfiles
  • Use tini or dumb-init as PID 1 in production containers
  • Set Docker shutdown timeout: docker stop -t 30 or stop_grace_period: 30s in compose
  • Test graceful shutdown: docker stop and verify clean exit code 0
  • Add health check that fails during shutdown to remove from load balancer
  • Monitor process_residence_memory_bytes dropping to zero during shutdown
  • Use Kubernetes terminationGracePeriodSeconds matching your shutdown timeout