Introduction

Go's context.WithTimeout and context.WithDeadline are used to propagate cancellation and set timeouts across API calls, database queries, and microservice communication. When the downstream operation takes longer than the deadline, context deadline exceeded is returned. In distributed systems, this cascades through the call chain if each service sets its own timeout without accounting for upstream deadlines.

Symptoms

  • context deadline exceeded from http.Client, database driver, or gRPC call
  • request canceled (Client.Timeout exceeded while awaiting headers)
  • Cascading timeouts across multiple services
  • Error occurs only under load, not in local testing
  • Partial responses where some downstream calls succeed and others timeout

```go ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel()

req, _ := http.NewRequestWithContext(ctx, "GET", "https://api.example.com/slow", nil) resp, err := http.DefaultClient.Do(req) if err != nil { // err = "Get \"https://api.example.com/slow\": context deadline exceeded" log.Println(err) } ```

Common Causes

  • Timeout too short for the actual operation latency
  • No budget propagation - each service uses full timeout independently
  • Downstream service degraded (slow DB, high GC pause, thread pool exhaustion)
  • Context deadline not propagated through gRPC/HTTP headers
  • Multiple sequential calls with same deadline consuming total budget

Step-by-Step Fix

  1. 1.Propagate deadline through the call chain:
  2. 2.```go
  3. 3.func HandleRequest(w http.ResponseWriter, r *http.Request) {
  4. 4.// Use the request's context - it already has the client's deadline
  5. 5.ctx := r.Context()

// Check if deadline exists if deadline, ok := ctx.Deadline(); ok { log.Printf("Remaining time: %v", time.Until(deadline)) }

result, err := callDownstream(ctx) // ... } ```

  1. 1.Budget deadlines for multiple downstream calls:
  2. 2.```go
  3. 3.func ProcessOrder(ctx context.Context, orderID string) error {
  4. 4.deadline, ok := ctx.Deadline()
  5. 5.if !ok {
  6. 6.var cancel func()
  7. 7.ctx, cancel = context.WithTimeout(ctx, 30*time.Second)
  8. 8.defer cancel()
  9. 9.deadline, _ = ctx.Deadline()
  10. 10.}

remaining := time.Until(deadline) budget := remaining / 3 // Split among 3 calls

// Call 1: validate inventory ctx1, cancel1 := context.WithTimeout(ctx, budget) defer cancel1() if err := checkInventory(ctx1, orderID); err != nil { return err }

// Call 2: process payment ctx2, cancel2 := context.WithTimeout(ctx, budget) defer cancel2() if err := processPayment(ctx2, orderID); err != nil { return err }

// Call 3: send confirmation ctx3, cancel3 := context.WithTimeout(ctx, budget) defer cancel3() return sendConfirmation(ctx3, orderID) } ```

  1. 1.Use context.WithTimeout for individual calls:
  2. 2.```go
  3. 3.func fetchWithTimeout(url string, timeout time.Duration) ([]byte, error) {
  4. 4.ctx, cancel := context.WithTimeout(context.Background(), timeout)
  5. 5.defer cancel()

req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) if err != nil { return nil, err }

resp, err := http.DefaultClient.Do(req) if err != nil { if ctx.Err() == context.DeadlineExceeded { return nil, fmt.Errorf("request to %s timed out after %v", url, timeout) } return nil, err } defer resp.Body.Close() return io.ReadAll(resp.Body) } ```

  1. 1.Propagate deadline through gRPC metadata:
  2. 2.```go
  3. 3.import "google.golang.org/grpc/metadata"

func clientInterceptor(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error { if deadline, ok := ctx.Deadline(); ok { // Send remaining timeout to downstream service md := metadata.Pairs("x-deadline", time.Until(deadline).String()) ctx = metadata.NewOutgoingContext(ctx, md) } return invoker(ctx, method, req, reply, cc, opts...) } ```

Prevention

  • Always set reasonable timeouts: 5s for internal APIs, 30s for external APIs
  • Use otel or tracing to visualize where time is spent across services
  • Implement circuit breaker pattern with sony/gobreaker
  • Monitor context deadline exceeded rate as an SLO indicator
  • Set gateway-level timeouts that are shorter than client timeouts
  • Use context.WithTimeoutCause (Go 1.21+) for better error attribution