Introduction

When Kubernetes sends SIGTERM to a Go container during pod termination, the application must gracefully shut down -- finish in-flight requests, close database connections, and flush buffered logs. If the Go process does not handle SIGTERM, it is killed after the termination grace period (default 30 seconds) with SIGKILL, causing dropped connections, lost log data, and potential data corruption. A common mistake is registering the signal handler incorrectly or blocking on signal reception in a way that prevents the handler from executing.

Symptoms

Container logs show abrupt termination:

bash
kubectl logs my-pod --previous
# No shutdown messages - process was killed by SIGKILL

Or the application receives SIGTERM but does not respond:

``` # In one terminal $ ./myapp & [1] 12345

# In another terminal $ kill -TERM 12345 # Application continues running, no shutdown messages # Only killed after kill -9 ```

Kubernetes events show:

bash
Warning  Killing    30s   kubelet  Stopping container myapp
Warning  FailedStop 0s    kubelet  Container did not stop within 30 seconds

Common Causes

  • signal.Notify not called: The program never registers interest in SIGTERM
  • Blocking on signal channel without buffer: Unbuffered signal channel requires a goroutine to read; if the reader is busy, the signal is dropped
  • SIGTERM handler blocking main goroutine: The handler tries to do cleanup that depends on a server that has already stopped
  • Using syscall.SIGTERM instead of os.Interrupt: On some platforms, the signal constants differ
  • Goroutine leak preventing shutdown: Background goroutines do not exit when context is cancelled
  • Docker using shell form ENTRYPOINT: Shell does not forward signals to child process (PID 1 issue)

Step-by-Step Fix

Step 1: Use signal.NotifyContext for clean shutdown

```go package main

import ( "context" "log" "net/http" "os" "os/signal" "syscall" "time" )

func main() { // Create context that cancels on SIGTERM or SIGINT ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT, ) defer stop()

srv := &http.Server{ Addr: ":8080", Handler: mux(), }

// Start server in goroutine go func() { log.Printf("Server starting on :8080") if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed { log.Fatalf("Server failed: %v", err) } }()

// Wait for shutdown signal <-ctx.Done() log.Println("Shutdown signal received, draining connections...")

// Graceful shutdown with timeout shutdownCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second) defer cancel()

if err := srv.Shutdown(shutdownCtx); err != nil { log.Fatalf("Server forced to shutdown: %v", err) }

log.Println("Server exited cleanly") } ```

Step 2: Handle Docker PID 1 signal forwarding

If using a shell wrapper, use exec to replace the shell with the Go binary:

```dockerfile # WRONG - shell does not forward signals CMD ./myapp

# CORRECT - exec replaces shell, Go process becomes PID 1 CMD ["./myapp"]

# Or if you need a shell wrapper: CMD ["sh", "-c", "exec ./myapp"] ```

Or use tini as init process:

dockerfile
RUN apt-get update && apt-get install -y tini
ENTRYPOINT ["tini", "--"]
CMD ["./myapp"]

Step 3: Graceful database and resource cleanup

```go func main() { ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM) defer stop()

db, err := sql.Open("postgres", dsn) if err != nil { log.Fatal(err) }

// Pass ctx to your application app := NewApp(db) go app.Run(ctx)

<-ctx.Done()

// Cleanup in order log.Println("Closing database connections...") db.Close()

log.Println("Flushing buffered logs...") logger.Sync()

log.Println("Shutdown complete") } ```

Prevention

  • Always use signal.NotifyContext rather than manual signal.Notify + channel
  • Set Kubernetes terminationGracePeriodSeconds to match your shutdown time needs (default 30s)
  • Add a preStop hook in Kubernetes to remove the pod from service endpoints before SIGTERM
  • Test shutdown behavior locally: kill -TERM $(pidof myapp) and verify clean exit
  • Use trap in shell scripts to forward signals: trap 'kill -TERM $PID' TERM INT
  • Monitor container restart reasons in Kubernetes: kubectl get pods -o wide and check the REASON column