Introduction

Go context deadline exceeded errors occur when a context.Context reaches its deadline before an operation completes, causing the context to be cancelled with context.DeadlineExceeded error. This is Go's primary mechanism for timeout handling and graceful cancellation across goroutines, HTTP handlers, database queries, and RPC calls. When contexts timeout, operations are aborted, resources are cleaned up, and cascading cancellations propagate through the call stack. The fix requires understanding context propagation patterns, appropriate timeout configuration for different operation types, goroutine lifecycle management, and proper error handling. This guide provides production-proven troubleshooting for Go context scenarios including HTTP client timeouts, database query timeouts, gRPC deadlines, goroutine leaks, and monitoring strategies.

Symptoms

  • Application logs show context deadline exceeded errors
  • HTTP requests timeout before response is received
  • Database queries cancelled mid-execution
  • gRPC calls fail with DeadlineExceeded status
  • Goroutines continue running after context cancellation (goroutine leaks)
  • Request tracing shows operations timing out at consistent thresholds
  • Error rate increases under load as operations take longer
  • Child goroutines not receiving parent context cancellation

Common Causes

  • HTTP client timeout set too low for upstream latency
  • Database query timeout exceeded for complex queries
  • gRPC deadline too short for service chain latency
  • Context not propagated to child goroutines
  • Goroutines not checking ctx.Done() channel
  • Resource contention causing operations to slow down
  • Downstream service degradation increasing response times
  • Network latency spikes exceeding timeout thresholds
  • Missing connection pooling causing slow connection establishment
  • Context timeout set at request level, not operation level

Step-by-Step Fix

### 1. Confirm context deadline diagnosis

Check error logs and stack traces:

```go // Check error type in logs if err != nil { if errors.Is(err, context.DeadlineExceeded) { log.Error("operation timed out", "error", err) } if errors.Is(err, context.Canceled) { log.Error("operation was cancelled", "error", err) } }

// Add context error wrapping for better debugging if err != nil { return fmt.Errorf("fetching user %d: %w", userID, err) }

// Typical error output: // fetching user 12345: context deadline exceeded ```

Trace context propagation:

```go // Add logging to trace context flow func handler(w http.ResponseWriter, r *http.Request) { ctx := r.Context() log.Info("handler started", "path", r.URL.Path)

result, err := doWork(ctx) if err != nil { log.Error("handler failed", "error", err) http.Error(w, err.Error(), http.StatusGatewayTimeout) return }

log.Info("handler completed") json.NewEncoder(w).Encode(result) }

func doWork(ctx context.Context) (Result, error) { log.Info("doWork started") defer log.Info("doWork completed")

// Check if context already cancelled select { case <-ctx.Done(): return Result{}, ctx.Err() default: }

// Do actual work with context return actualWork(ctx) } ```

### 2. Check HTTP client timeout configuration

HTTP clients need explicit timeout configuration:

```go // WRONG: No timeout configuration client := &http.Client{} resp, err := client.Get(url)

// CORRECT: Configure timeouts client := &http.Client{ Timeout: 30 * time.Second, // Overall request timeout Transport: &http.Transport{ DialContext: (&net.Dialer{ Timeout: 10 * time.Second, // Connection timeout KeepAlive: 30 * time.Second, }).DialContext, MaxIdleConns: 100, IdleConnTimeout: 90 * time.Second, TLSHandshakeTimeout: 10 * time.Second, ExpectContinueTimeout: 1 * time.Second, }, }

// Or use context with timeout for per-request control ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel()

req, err := http.NewRequestWithContext(ctx, "GET", url, nil) if err != nil { return nil, err }

resp, err := client.Do(req) if err != nil { if errors.Is(err, context.DeadlineExceeded) { log.Error("request timed out", "url", url) } return nil, err } defer resp.Body.Close() ```

Per-endpoint timeout configuration:

```go type Client struct { httpClient *http.Client fastTimeout time.Duration slowTimeout time.Duration }

func NewClient() *Client { return &Client{ httpClient: &http.Client{ Timeout: 60 * time.Second, Transport: &http.Transport{ DialContext: (&net.Dialer{ Timeout: 10 * time.Second, }).DialContext, }, }, fastTimeout: 5 * time.Second, // For simple lookups slowTimeout: 30 * time.Second, // For complex operations } }

func (c *Client) FastRequest(ctx context.Context, url string) (*http.Response, error) { ctx, cancel := context.WithTimeout(ctx, c.fastTimeout) defer cancel() return c.doRequest(ctx, url) }

func (c *Client) SlowRequest(ctx context.Context, url string) (*http.Response, error) { ctx, cancel := context.WithTimeout(ctx, c.slowTimeout) defer cancel() return c.doRequest(ctx, url) } ```

### 3. Configure database query timeouts

Database operations need context-aware timeouts:

```go // database/sql with context db, err := sql.Open("postgres", dsn) if err != nil { return nil, err }

// Set connection pool parameters db.SetMaxOpenConns(25) db.SetMaxIdleConns(25) db.SetConnMaxLifetime(5 * time.Minute)

// Query with context timeout func getUser(ctx context.Context, db *sql.DB, userID int64) (*User, error) { // Set operation-level timeout ctx, cancel := context.WithTimeout(ctx, 10*time.Second) defer cancel()

row := db.QueryRowContext(ctx, "SELECT id, name, email FROM users WHERE id = $1", userID)

var user User err := row.Scan(&user.ID, &user.Name, &user.Email) if err != nil { if errors.Is(err, context.DeadlineExceeded) { log.Error("query timed out", "userID", userID) } return nil, err } return &user, nil }

// Transaction with context func transferMoney(ctx context.Context, db *sql.DB, from, to int64, amount decimal) error { ctx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel()

tx, err := db.BeginTx(ctx, nil) if err != nil { return err } defer tx.Rollback()

// All operations use tx with context _, err = tx.ExecContext(ctx, "UPDATE accounts SET balance = balance - $1 WHERE id = $2", amount, from) if err != nil { return err }

_, err = tx.ExecContext(ctx, "UPDATE accounts SET balance = balance + $1 WHERE id = $2", amount, to) if err != nil { return err }

return tx.Commit() } ```

GORM with context:

```go // GORM queries with context func GetUser(ctx context.Context, db *gorm.DB, userID uint) (*User, error) { ctx, cancel := context.WithTimeout(ctx, 10*time.Second) defer cancel()

var user User err := db.WithContext(ctx).First(&user, userID).Error if err != nil { return nil, err } return &user, nil }

// Batch operations with context func ProcessUsers(ctx context.Context, db *gorm.DB) error { var batch []User offset := 0 limit := 1000

for { // Check context before each batch select { case <-ctx.Done(): return ctx.Err() default: }

err := db.WithContext(ctx). Limit(limit). Offset(offset). Find(&batch).Error if err != nil { return err }

if len(batch) == 0 { break }

// Process batch for _, user := range batch { if err := processUser(ctx, user); err != nil { return err } }

offset += limit } return nil } ```

### 4. Configure gRPC deadlines

gRPC uses context for deadline propagation:

```go // Client with deadline conn, err := grpc.Dial(address, grpc.WithInsecure()) if err != nil { return nil, err }

client := pb.NewMyServiceClient(conn)

// Set deadline for RPC call ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel()

resp, err := client.MyMethod(ctx, &pb.Request{Data: "value"}) if err != nil { if status.Code(err) == codes.DeadlineExceeded { log.Error("gRPC deadline exceeded") } return nil, err }

// Per-RPC timeout configuration type GRPCClient struct { conn *grpc.ClientConn client pb.MyServiceClient }

func (c *GRPCClient) FastCall(ctx context.Context, req *pb.Request) (*pb.Response, error) { ctx, cancel := context.WithTimeout(ctx, 5*time.Second) defer cancel() return c.client.MyMethod(ctx, req) }

func (c *GRPCClient) SlowCall(ctx context.Context, req *pb.Request) (*pb.Response, error) { ctx, cancel := context.WithTimeout(ctx, 60*time.Second) defer cancel() return c.client.MyMethod(ctx, req) }

// Server-side deadline handling func (s *Server) MyMethod(ctx context.Context, req *pb.Request) (*pb.Response, error) { // Check deadline at start deadline, ok := ctx.Deadline() if ok { log.Info("request deadline", "deadline", deadline) }

// Check context periodically during long operations for i := 0; i < 100; i++ { select { case <-ctx.Done(): return nil, ctx.Err() default: }

// Do work processItem(i) }

return &pb.Response{Result: "success"}, nil } ```

### 5. Propagate context to goroutines

Ensure child goroutines receive parent context:

```go // WRONG: Goroutine doesn't receive context func handler(w http.ResponseWriter, r *http.Request) { go doWork() // No context! Goroutine continues if request cancelled w.Write([]byte("ok")) }

// CORRECT: Pass context to goroutine func handler(w http.ResponseWriter, r *http.Request) { ctx := r.Context() go doWork(ctx) // Goroutine can be cancelled w.Write([]byte("ok")) }

func doWork(ctx context.Context) { select { case <-ctx.Done(): log.Info("work cancelled") return case result := <-doActualWork(): log.Info("work completed", "result", result) } }

// Worker pool with context func StartWorkerPool(ctx context.Context, numWorkers int, jobs <-chan Job) { for i := 0; i < numWorkers; i++ { go func(id int) { for { select { case <-ctx.Done(): log.Info("worker shutting down", "id", id) return case job, ok := <-jobs: if !ok { return } processJob(ctx, job) } } }(i) } }

// Usage ctx, cancel := context.WithCancel(context.Background()) defer cancel()

jobs := make(chan Job, 100) StartWorkerPool(ctx, 10, jobs)

// Submit jobs jobs <- Job{Data: "process this"}

// Shutdown (workers receive cancellation) cancel() ```

### 6. Handle goroutine leaks

Ensure goroutines exit on context cancellation:

```go // WRONG: Goroutine leak func doWork(ctx context.Context) chan Result { results := make(chan Result) go func() { // Never exits if ctx is cancelled! result := heavyComputation() results <- result // Blocks forever if no one listening }() return results }

// CORRECT: Check context and use buffered channel func doWork(ctx context.Context) chan Result { results := make(chan Result, 1) // Buffered to prevent blocking go func() { defer close(results)

select { case <-ctx.Done(): return default: }

result := heavyComputation()

select { case results <- result: case <-ctx.Done(): return } }() return results }

// Pattern: Context-aware worker with cleanup type Worker struct { ctx context.Context cancel context.CancelFunc }

func NewWorker() *Worker { ctx, cancel := context.WithCancel(context.Background()) return &Worker{ctx: ctx, cancel: cancel} }

func (w *Worker) Start() { go w.run() }

func (w *Worker) Stop() { w.cancel() // Signal goroutine to exit }

func (w *Worker) run() { ticker := time.NewTicker(time.Second) defer ticker.Stop()

for { select { case <-w.ctx.Done(): log.Info("worker stopping") return case <-ticker.C: w.doPeriodicWork() } } } ```

### 7. Implement retry with backoff

Add retry logic with context awareness:

```go func doWithRetry(ctx context.Context, operation func() error) error { var lastErr error

for attempt := 1; attempt <= 3; attempt++ { // Check context before each attempt select { case <-ctx.Done(): return ctx.Err() default: }

lastErr = operation() if lastErr == nil { return nil }

// Don't retry on context errors if errors.Is(lastErr, context.DeadlineExceeded) || errors.Is(lastErr, context.Canceled) { return lastErr }

// Exponential backoff with jitter backoff := time.Duration(attempt*attempt) * 100 * time.Millisecond jitter := time.Duration(rand.Int63n(int64(backoff / 2))) sleepTime := backoff + jitter

log.Info("retrying", "attempt", attempt, "error", lastErr, "sleep", sleepTime)

// Sleep with context awareness select { case <-ctx.Done(): return ctx.Err() case <-time.After(sleepTime): } }

return fmt.Errorf("operation failed after %d attempts: %w", 3, lastErr) }

// Usage with HTTP client func fetchWithRetry(ctx context.Context, client *http.Client, url string) ([]byte, error) { var data []byte err := doWithRetry(ctx, func() error { req, err := http.NewRequestWithContext(ctx, "GET", url, nil) if err != nil { return err }

resp, err := client.Do(req) if err != nil { return err } defer resp.Body.Close()

if resp.StatusCode >= 500 { return fmt.Errorf("server error: %d", resp.StatusCode) }

data, err = io.ReadAll(resp.Body) return err })

return data, err } ```

### 8. Set appropriate timeout values

Timeout guidelines by operation type:

```go // Timeout configuration constants const ( // Fast operations (cache lookups, simple queries) FastTimeout = 5 * time.Second

// Normal operations (database queries, HTTP calls) NormalTimeout = 30 * time.Second

// Slow operations (batch processing, file operations) SlowTimeout = 2 * time.Minute

// Very slow operations (report generation, data export) VerySlowTimeout = 10 * time.Minute )

// HTTP handler with timeout middleware func WithTimeout(timeout time.Duration) func(http.Handler) http.Handler { return func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { ctx, cancel := context.WithTimeout(r.Context(), timeout) defer cancel()

next.ServeHTTP(w, r.WithContext(ctx)) }) } }

// Usage in router mux := http.NewServeMux()

// Apply different timeouts per route mux.Handle("/api/fast", WithTimeout(FastTimeout)(http.HandlerFunc(fastHandler))) mux.Handle("/api/normal", WithTimeout(NormalTimeout)(http.HandlerFunc(normalHandler))) mux.Handle("/api/slow", WithTimeout(SlowTimeout)(http.HandlerFunc(slowHandler)))

// Default timeout for all other routes handler := WithTimeout(NormalTimeout)(mux) http.ListenAndServe(":8080", handler) ```

Timeout hierarchy (child should be <= parent):

```go // Request-level timeout func handler(w http.ResponseWriter, r *http.Request) { requestCtx, requestCancel := context.WithTimeout(r.Context(), 30*time.Second) defer requestCancel()

// Database operation (shorter timeout) dbCtx, dbCancel := context.WithTimeout(requestCtx, 10*time.Second) defer dbCancel()

user, err := getUser(dbCtx, db, userID) if err != nil { http.Error(w, err.Error(), http.StatusGatewayTimeout) return }

// External API call (remaining timeout) apiCtx, apiCancel := context.WithTimeout(requestCtx, 15*time.Second) defer apiCancel()

data, err := fetchFromAPI(apiCtx, user.ID) if err != nil { http.Error(w, err.Error(), http.StatusGatewayTimeout) return }

json.NewEncoder(w).Encode(data) } ```

### 9. Monitor context cancellations

Add observability for context issues:

```go // Prometheus metrics for context cancellations var ( contextCancellationTotal = promauto.NewCounterVec( prometheus.CounterOpts{ Name: "context_cancellations_total", Help: "Total number of context cancellations", }, []string{"reason"}, // "deadline_exceeded", "canceled" )

operationDuration = promauto.NewHistogramVec( prometheus.HistogramOpts{ Name: "operation_duration_seconds", Help: "Duration of operations", Buckets: prometheus.ExponentialBuckets(0.01, 2, 10), }, []string{"operation", "status"}, ) )

// Instrumented operation func instrumentedOperation(ctx context.Context, name string, op func(ctx context.Context) error) error { start := time.Now() err := op(ctx) duration := time.Since(start)

var status string if err != nil { if errors.Is(err, context.DeadlineExceeded) { status = "deadline_exceeded" contextCancellationTotal.WithLabelValues("deadline_exceeded").Inc() } else if errors.Is(err, context.Canceled) { status = "canceled" contextCancellationTotal.WithLabelValues("canceled").Inc() } else { status = "error" } } else { status = "success" }

operationDuration.WithLabelValues(name, status).Observe(duration.Seconds())

return err } ```

Structured logging for context:

```go // Add context information to logs type contextKey string const requestIDKey contextKey = "request_id"

func WithRequestID(ctx context.Context, id string) context.Context { return context.WithValue(ctx, requestIDKey, id) }

func getRequestID(ctx context.Context) string { if id, ok := ctx.Value(requestIDKey).(string); ok { return id } return "" }

// Log with context func logOperation(ctx context.Context, message string, err error) { log := slog.With( "request_id", getRequestID(ctx), "message", message, )

if deadline, ok := ctx.Deadline(); ok { log = log.With("deadline", deadline) log = log.With("time_remaining", time.Until(deadline)) }

if err != nil { log.Error(message, "error", err) } else { log.Info(message) } } ```

### 10. Handle cascading cancellations

Design for graceful cancellation propagation:

```go // Service with multiple dependent calls func ProcessOrder(ctx context.Context, orderID string) error { // Create child contexts for each phase validateCtx, validateCancel := context.WithTimeout(ctx, 5*time.Second) defer validateCancel()

if err := validateOrder(validateCtx, orderID); err != nil { return fmt.Errorf("validation failed: %w", err) }

reserveCtx, reserveCancel := context.WithTimeout(ctx, 10*time.Second) defer reserveCancel()

if err := reserveInventory(reserveCtx, orderID); err != nil { return fmt.Errorf("reservation failed: %w", err) }

chargeCtx, chargeCancel := context.WithTimeout(ctx, 15*time.Second) defer chargeCancel()

if err := chargePayment(chargeCtx, orderID); err != nil { // Compensating transaction needed rollbackCtx, rollbackCancel := context.WithTimeout(ctx, 10*time.Second) defer rollbackCancel()

// Try to rollback reservation if rollbackErr := releaseInventory(rollbackCtx, orderID); rollbackErr != nil { log.Error("rollback failed", "error", rollbackErr) }

return fmt.Errorf("payment failed: %w", err) }

return nil }

// Fan-out pattern with context func FanOut(ctx context.Context, urls []string) ([]Result, error) { results := make([]Result, len(urls)) errors := make(chan error, len(urls))

// Create child context for all workers workerCtx, cancel := context.WithCancel(ctx) defer cancel()

var wg sync.WaitGroup for i, url := range urls { wg.Add(1) go func(idx int, u string) { defer wg.Done()

result, err := fetchURL(workerCtx, u) if err != nil { errors <- err return }

results[idx] = result }(i, url) }

// Wait for completion or cancellation done := make(chan struct{}) go func() { wg.Wait() close(done) }()

select { case <-ctx.Done(): cancel() // Signal all workers to stop return nil, ctx.Err() case err := <-errors: cancel() // First error cancels all return nil, err case <-done: return results, nil } } ```

Prevention

  • Always pass context to functions that perform I/O or blocking operations
  • Set explicit timeouts for HTTP clients, database queries, and RPC calls
  • Use context.WithTimeout for individual operations, not just requests
  • Check ctx.Done() in long-running loops and goroutines
  • Use buffered channels to prevent goroutine blocking
  • Implement retry logic with exponential backoff and context awareness
  • Propagate request context to all downstream calls
  • Log context deadline information for debugging
  • Monitor context cancellation rates and operation durations
  • Document timeout requirements for each service endpoint
  • **context canceled**: Operation was cancelled before deadline
  • **http: Server.Timeout**: HTTP server request timeout
  • **i/o timeout**: Network operation timeout
  • **connection timeout**: TCP connection establishment timeout
  • **read timeout**: Socket read operation timeout