Introduction

Go's pprof CPU profiling is essential for identifying performance bottlenecks in production services. However, a common issue is that the generated profile contains no or minimal samples, making it impossible to identify hot paths. This happens because CPU profiling samples goroutine execution at 100Hz (100 samples per second per goroutine), so short-lived profiles, idle processes, or incorrectly started/stopped profilers produce insufficient data. The profile appears to work -- it generates a file -- but the file contains no meaningful samples.

Symptoms

The profile file is generated but has no data:

bash
go tool pprof -top cpu.prof
File: myapp
Type: cpu
Time: Mar 15, 2024 at 10:23am (UTC)
Duration: 300ms, Total samples = 0
Showing nodes accounting for 0, 0% of 0 total

Or via the HTTP endpoint:

bash
curl -o cpu.prof http://localhost:6060/debug/pprof/profile?seconds=1
go tool pprof -top cpu.prof
# Duration: 1.00s, Total samples = 0

Or the profile only shows runtime functions:

bash
Showing nodes accounting for 50, 100% of 50 total
      50   100%   100%         50   100%  runtime.pthread_cond_wait

Common Causes

  • Profiling duration too short: Less than 1 second of profiling may capture no samples at 100Hz
  • Application is idle during profiling: No CPU work happening during the profile window
  • Profile started but not stopped: pprof.StartCPUProfile called without matching pprof.StopCPUProfile
  • HTTP profile endpoint blocked: Middleware or firewall blocks access to /debug/pprof/profile
  • Container CPU limits: cgroup CPU quota limits prevent the profiler from sampling
  • Profile file written to wrong location: File written but overwritten by subsequent profiling

Step-by-Step Fix

Step 1: Correct CPU profiling with proper duration

```go import ( "os" "runtime/pprof" "time" )

func ProfileCPU(duration time.Duration, outputPath string) error { f, err := os.Create(outputPath) if err != nil { return fmt.Errorf("create profile file: %w", err) } defer f.Close()

if err := pprof.StartCPUProfile(f); err != nil { return fmt.Errorf("start CPU profile: %w", err) }

// Wait for the specified duration time.Sleep(duration)

pprof.StopCPUProfile() log.Printf("CPU profile written to %s", outputPath) return nil }

// Usage: profile for at least 10 seconds for meaningful data ProfileCPU(10*time.Second, "/tmp/cpu.prof") ```

Step 2: Use the HTTP profile endpoint correctly

```go import ( _ "net/http/pprof" // Registers pprof handlers "net/http" )

func main() { // Start pprof server on a separate port go func() { log.Println("pprof server starting on :6061") log.Println(http.ListenAndServe("localhost:6061", nil)) }()

// Your application logic... } ```

Then collect a profile:

```bash # Interactive pprof - collects 30 seconds of CPU profile automatically go tool pprof http://localhost:6061/debug/pprof/profile?seconds=30

# Or download first, analyze later curl -o cpu.prof "http://localhost:6061/debug/pprof/profile?seconds=30" go tool pprof -http=:8080 cpu.prof ```

Step 3: Ensure the application is doing work during profiling

```go func main() { // Start profiling f, _ := os.Create("cpu.prof") pprof.StartCPUProfile(f) defer pprof.StopCPUProfile() defer f.Close()

// Run the workload that you want to profile runBenchmark() // This must do actual CPU work

// If runBenchmark returns too quickly, the profile will be empty // Add a minimum duration check } ```

Step 4: Check container CPU settings

In Docker or Kubernetes, CPU limits can interfere with profiling:

yaml
# Kubernetes - ensure CPU request is sufficient
resources:
  requests:
    cpu: "500m"    # At least 0.5 CPU for profiling
  limits:
    cpu: "1000m"   # Do not set limits too low

If the container is CPU-throttled, the profiler cannot sample:

bash
# Check if the container is being throttled
cat /sys/fs/cgroup/cpu/cpu.stat
# Look for nr_throttled - if high, CPU limits are too restrictive

Prevention

  • Always profile for at least 10-30 seconds in production
  • Ensure the endpoint or function being profiled is actively receiving traffic
  • Use go tool pprof -top cpu.prof immediately after collection to verify sample count
  • Add a health check that verifies pprof.StartCPUProfile returns no error
  • In CI, add a test that generates a profile and asserts Total samples > 0
  • Use block and mutex profiles alongside CPU: curl http://localhost:6061/debug/pprof/block