Introduction

Pandas loads entire DataFrames into memory, which means a 2GB CSV file can consume 10-20GB of RAM after parsing (due to object dtype overhead, index creation, and intermediate copies during operations). When memory exceeds system limits, the Linux OOM killer terminates the Python process, or Python raises MemoryError. This is a fundamental limitation of Pandas' in-memory design, not a bug. The solution involves reducing memory footprint through dtype optimization, processing data in chunks, or switching to out-of-core libraries that do not require all data in RAM.

Symptoms

bash
MemoryError: Unable to allocate 8.23 GiB for an array with shape (1099511627776,) and data type float64

Or OOM kill:

bash
Killed
# dmesg shows:
# Out of memory: Killed process 12345 (python3) total-vm:18432000kB

Or during merge:

bash
MemoryError: Unable to allocate 15.4 GiB for an array with shape (2073600000,) and data type int64

Common Causes

  • All string columns as object dtype: Each string stored as Python object with overhead
  • Merge creating cartesian product: Duplicate keys cause exponential memory growth
  • Intermediate copies during operations: df.copy(), groupby, sort create temporary arrays
  • Reading entire file at once: read_csv() loads all data into memory
  • Wide tables with many columns: Each column adds overhead
  • Not releasing references: Old DataFrames still referenced while creating new ones

Step-by-Step Fix

Step 1: Optimize dtypes during load

```python import pandas as pd

def optimize_dtypes(df): """Reduce DataFrame memory by optimizing column types.""" for col in df.columns: col_type = df[col].dtype

if pd.api.types.is_integer_dtype(col_type): c_min = df[col].min() c_max = df[col].max() if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max: df[col] = df[col].astype(np.int8) elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max: df[col] = df[col].astype(np.int16) elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max: df[col] = df[col].astype(np.int32)

elif pd.api.types.is_float_dtype(col_type): df[col] = pd.to_numeric(df[col], downcast='float')

elif pd.api.types.is_object_dtype(col_type): # Use categorical for low-cardinality string columns if df[col].nunique() / len(df) < 0.5: df[col] = df[col].astype('category')

return df

# Usage df = pd.read_csv('large_file.csv') df = optimize_dtypes(df) # Typical reduction: 50-80% memory ```

Step 2: Process data in chunks

```python def process_large_csv(filepath, chunksize=100_000): """Process a large CSV without loading it all into memory.""" results = []

for chunk in pd.read_csv(filepath, chunksize=chunksize): chunk = optimize_dtypes(chunk) # Process each chunk result = chunk.groupby('category')['value'].sum() results.append(result)

# Combine results (much smaller than original data) final = pd.concat(results).groupby(level=0).sum() return final

# Usage result = process_large_csv('data_10gb.csv', chunksize=100_000) ```

Step 3: Use out-of-core processing with Dask or Polars

```python # Option A: Dask - Pandas-compatible API import dask.dataframe as dd

ddf = dd.read_csv('large_file.csv') result = ddf.groupby('category')['value'].sum().compute()

# Option B: Polars - faster and more memory-efficient import polars as pl

df = pl.scan_csv('large_file.csv') # Lazy, doesn't load data yet result = df.group_by('category').agg(pl.col('value').sum()).collect()

# Streaming for files larger than RAM result = df.group_by('category').agg(pl.col('value').sum()).collect(streaming=True) ```

Prevention

  • Use df.memory_usage(deep=True) to identify memory-hungry columns
  • Convert string columns to 'category' dtype when cardinality is below 50%
  • Use chunked processing for CSV files larger than 1/4 of available RAM
  • Delete intermediate DataFrames with del df and call gc.collect()
  • Prefer Polars or Dask for datasets exceeding available memory
  • Monitor memory with tracemalloc during development to find hotspots
  • Set memory limits in container environments to get MemoryError instead of OOM kill