Introduction

Go's net/http package maintains a connection pool through the http.Transport to reuse TCP connections across requests. When this pool is exhausted -- typically because response bodies are not closed, idle connection limits are too low, or connections are leaked -- new requests block waiting for an available connection, eventually timing out with context deadline exceeded or no free connections available. This error is insidious in production because it appears gradually under increasing load and manifests as intermittent timeouts rather than a clean failure.

Symptoms

bash
Get "https://api.example.com/data": dial tcp 10.0.1.50:443: connect: connection refused

Or the more specific transport error:

bash
Get "https://api.example.com/data": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Checking active connections shows the leak:

bash
$ ss -tnp | grep :443 | wc -l
847

The Go runtime shows goroutine growth:

bash
goroutine profile: total 1523
892 in state select, net/http.(*persistConn).roundTrip

Common Causes

  • Not closing response body: resp.Body must be closed for the connection to be reused; forgetting this leaks connections
  • Default MaxIdleConnsPerHost is 2: The transport default only keeps 2 idle connections per host, discarding the rest
  • Creating a new http.Client per request: Each client has its own transport and pool, preventing connection reuse
  • Response body not fully consumed: Reading only part of the body and closing prevents connection reuse
  • Slowloris from downstream server: Connections held open by a slow downstream server consume pool slots
  • TIME_WAIT accumulation: High request rate to many different hosts creates TIME_WAIT sockets

Step-by-Step Fix

Step 1: Always close response body with defer

```go // WRONG - body not closed on error path resp, err := http.Get("https://api.example.com/data") if err != nil { return err } defer resp.Body.Close() // Only runs if err is nil data, err := io.ReadAll(resp.Body)

// CORRECT - body closed in all cases resp, err := http.Get("https://api.example.com/data") if err != nil { return err } defer resp.Body.Close() data, err := io.ReadAll(resp.Body) if err != nil { return err // Body still closed by defer } ```

Step 2: Configure transport with proper connection limits

```go var httpClient = &http.Client{ Timeout: 10 * time.Second, Transport: &http.Transport{ MaxIdleConns: 100, // Total idle connections MaxIdleConnsPerHost: 20, // Per-host idle connections (default is 2!) IdleConnTimeout: 90 * time.Second, // Close idle connections after this MaxConnsPerHost: 50, // Hard limit per host (0 = unlimited) }, }

func fetchData(ctx context.Context, url string) ([]byte, error) { req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) if err != nil { return nil, err }

resp, err := httpClient.Do(req) if err != nil { return nil, err } defer resp.Body.Close()

if resp.StatusCode != http.StatusOK { return nil, fmt.Errorf("unexpected status: %s", resp.Status) }

return io.ReadAll(resp.Body) } ```

The key fix: MaxIdleConnsPerHost: 20 instead of the default 2.

Step 3: Reuse a single http.Client

```go // Package-level singleton - do NOT create a new client per request var apiClient = &http.Client{ Timeout: 30 * time.Second, Transport: &http.Transport{ MaxIdleConnsPerHost: 20, IdleConnTimeout: 90 * time.Second, }, }

// Handler uses the shared client func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { data, err := fetchFromAPI(r.Context(), apiClient, "https://api.example.com/data") if err != nil { http.Error(w, err.Error(), http.StatusBadGateway) return } // ... } ```

Step 4: Monitor connection pool state

```go import "net/http/httptrace"

func fetchWithTracing(ctx context.Context, client *http.Client, url string) error { var gotConn, putIdleConn, waitDuration time.Duration

trace := &httptrace.ClientTrace{ GotConn: func(connInfo httptrace.GotConnInfo) { if connInfo.Reused { log.Printf("Reused connection to %s", connInfo.Conn.RemoteAddr()) } else { log.Printf("New connection to %s", connInfo.Conn.RemoteAddr()) } }, PutIdleConn: func(err error) { if err != nil { log.Printf("Failed to put connection idle: %v", err) } }, }

ctx = httptrace.WithClientTrace(ctx, trace) req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) resp, err := client.Do(req) if err != nil { return err } defer resp.Body.Close()

_ = putIdleConn _ = waitDuration _ = gotConn return nil } ```

Prevention

  • Always set MaxIdleConnsPerHost to match your concurrency level (not the default 2)
  • Use a single shared http.Client -- never create one per request
  • Always defer resp.Body.Close() immediately after checking the error
  • Set Client.Timeout to prevent indefinite blocking
  • Use go tool pprof -http=:8080 to monitor goroutine count for persistConn leaks
  • Add metrics for active connections and connection reuse rate in production monitoring