Introduction

filepath.Walk in Go has a subtle behavior where encountering a directory with permission denied (EACCES) causes the walk to skip that directory and all its contents without reporting an error to the caller -- unless the walk function explicitly handles the error. This means a file scanner, backup tool, or indexer can silently miss entire directory trees when running as a non-root user. In production, this leads to incomplete backups, missing index entries, or security audit gaps where restricted directories are never scanned.

Symptoms

A file scanner reports fewer files than expected:

go
var count int
filepath.Walk("/data", func(path string, info os.FileInfo, err error) error {
    if err != nil {
        // This error is printed but walk continues - directory contents are skipped
        log.Printf("error walking %s: %v", path, err)
        return nil  // Returning nil means "continue" but the directory is already skipped
    }
    if !info.IsDir() {
        count++
    }
    return nil
})
fmt.Printf("Found %d files\n", count)
// Output: Found 234 files (missing 567 files in /data/restricted/)

The error is logged but the directory tree is silently skipped:

bash
2024/03/15 10:23:45 error walking /data/restricted: lstat /data/restricted: permission denied

Common Causes

  • Walk running as non-root user: User does not have read permission on some directories
  • Returning nil on error: Returning nil from the walk function tells Walk to continue, but the denied directory's contents are already skipped
  • Not returning the error: Swallowing the error means the caller never knows about the permission issue
  • Using Walk instead of WalkDir: Walk calls os.Lstat on every file, which can fail for permission-denied entries
  • Docker volume mount permissions: Mounted volumes with different UID/GID cause unexpected permission errors

Step-by-Step Fix

Step 1: Handle permission errors explicitly

```go var skippedDirs []string

err := filepath.Walk("/data", func(path string, info os.FileInfo, err error) error { if err != nil { if os.IsPermission(err) { skippedDirs = append(skippedDirs, path) log.Printf("Skipping (permission denied): %s", path) return filepath.SkipDir } return err }

if !info.IsDir() { processFile(path) } return nil })

if err != nil { log.Fatalf("Walk failed: %v", err) }

if len(skippedDirs) > 0 { log.Printf("WARNING: %d directories skipped due to permission errors", len(skippedDirs)) for _, d := range skippedDirs { log.Printf(" - %s", d) } } ```

Step 2: Use filepath.WalkDir (Go 1.16+) for better performance

```go err := filepath.WalkDir("/data", func(path string, d fs.DirEntry, err error) error { if err != nil { if os.IsPermission(err) { log.Printf("Permission denied: %s", path) return fs.SkipDir } return err }

if !d.IsDir() { info, err := d.Info() if err != nil { log.Printf("Cannot stat %s: %v", path, err) return nil } processFile(path, info) } return nil }) ```

WalkDir is preferred because it avoids calling Lstat on every file -- DirEntry already provides the information needed for most use cases.

Step 3: Collect and report all errors

```go type WalkResult struct { Files []string SkippedDirs []string Errors []error }

func WalkAndCollect(root string) (*WalkResult, error) { result := &WalkResult{}

err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error { if err != nil { result.Errors = append(result.Errors, fmt.Errorf("%s: %w", path, err)) if os.IsPermission(err) { result.SkippedDirs = append(result.SkippedDirs, path) return fs.SkipDir } return err // Non-permission errors stop the walk }

if !d.IsDir() { result.Files = append(result.Files, path) } return nil })

return result, err } ```

Prevention

  • Always check for os.IsPermission(err) in the walk callback
  • Return fs.SkipDir (not nil) when you encounter a permission-denied directory
  • Use filepath.WalkDir instead of filepath.Walk for better performance
  • Report skipped directories to the caller so they are not silently missed
  • Run file scanners with appropriate permissions (e.g., as a dedicated service account with read access)
  • Use find /data -type f 2>/dev/null | wc -l as an independent verification of file count