Introduction

A fork bomb occurs when a process creates child processes that each create more children, exponentially consuming system resources. In Python multiprocessing, this can happen when worker functions themselves spawn processes, when pool size is not bounded, or when the fork start method duplicates all threads from the parent process.

Symptoms

  • System becomes unresponsive as process count explodes
  • OSError: [Errno 11] Resource temporarily unavailable (no more PIDs)
  • fork: retry: Resource temporarily unavailable
  • System log shows: cgroup: fork rejected by pids controller
  • dmesg shows: cgroup: fork rejected by pids controller in /user.slice
bash
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  ...
OSError: [Errno 11] Resource temporarily unavailable

Common Causes

  • Workers spawning additional processes without bounds
  • Using fork start method with threaded parent process
  • Pool processes creating their own pools recursively
  • Not setting if __name__ == '__main__' guard on Windows/macOS
  • Child processes inheriting parent's file descriptors and connections

Step-by-Step Fix

  1. 1.Always use the main guard:
  2. 2.```python
  3. 3.# WRONG - code at module level spawns processes immediately
  4. 4.from multiprocessing import Process
  5. 5.p = Process(target=worker) # Runs at import time!
  6. 6.p.start()

# CORRECT - guarded by __main__ check from multiprocessing import Process

def worker(): pass

if __name__ == '__main__': p = Process(target=worker) p.start() p.join() ```

  1. 1.Use spawn method instead of fork:
  2. 2.```python
  3. 3.import multiprocessing as mp

if __name__ == '__main__': mp.set_start_method('spawn') # Clean process, no inherited state pool = mp.Pool(processes=4) results = pool.map(process_item, data) pool.close() pool.join() ```

  1. 1.Limit pool size and tasks per child:
  2. 2.```python
  3. 3.import multiprocessing as mp

# Bound the number of processes num_workers = min(mp.cpu_count(), 8)

# maxtasksperchild prevents memory leaks and runaway processes with mp.Pool(processes=num_workers, maxtasksperchild=100) as pool: results = pool.map(heavy_computation, data_chunks) ```

  1. 1.Prevent workers from spawning child processes:
  2. 2.```python
  3. 3.import multiprocessing as mp

def worker(item): # Do NOT create Pool/Process inside worker result = process(item) # Single-threaded within worker return result

if __name__ == '__main__': with mp.Pool(4) as pool: pool.map(worker, data) ```

  1. 1.Set system-level process limits:
  2. 2.```bash
  3. 3.# Check current limits
  4. 4.ulimit -u # Max user processes

# Set a limit (temporary) ulimit -u 4096

# Set permanently in /etc/security/limits.d/99-limits.conf # python_user soft nproc 4096 # python_user hard nproc 8192 ```

  1. 1.Use concurrent.futures for safer process management:
  2. 2.```python
  3. 3.from concurrent.futures import ProcessPoolExecutor, as_completed

with ProcessPoolExecutor(max_workers=4) as executor: futures = {executor.submit(process_item, item): item for item in data} for future in as_completed(futures): try: result = future.result(timeout=300) except Exception as e: print(f"Task failed: {e}") ```

Prevention

  • Always use if __name__ == '__main__': guard
  • Set max_workers explicitly, never use unlimited
  • Use spawn start method on Linux to avoid fork-with-threads issues
  • Monitor process count during development: ps aux | wc -l
  • Add cgroup limits in containerized environments
  • Use maxtasksperchild to recycle workers periodically
  • Test with small datasets before running at scale