Introduction Memcached uses an LRU (Least Recently Used) eviction policy when memory is full. Under memory pressure, frequently accessed keys can be evicted if they happen to be at the tail of the LRU list, causing cache stampedes and database overload. This is particularly problematic for hot keys that are accessed thousands of times per second.

Symptoms - `STAT evictions` increasing steadily in memcached stats - Hot keys being evicted and immediately re-fetched from database - Database load spiking as evicted keys cause cache misses - `STAT reclaimed` increasing, indicating expired items being reclaimed - Cache hit rate dropping despite steady traffic patterns

Common Causes - Memcached memory size too small for the working set - LRU tail being populated by one-time large objects evicting hot small keys - No TTL differentiation between hot and cold keys - Memory fragmentation reducing effective cache capacity - Large object insertions pushing hot keys out of memory

Step-by-Step Fix 1. **Check eviction statistics": ```bash echo "stats" | nc localhost 11211 | grep -E "evictions|reclaimed|curr_items|bytes|limit_maxbytes" # Example: # STAT evictions 12345 # STAT reclaimed 5678 # STAT curr_items 456789 # STAT bytes 3800000000 # STAT limit_maxbytes 4294967296 ```

  1. 1.**Enable LRU crawler to proactively reclaim expired items":
  2. 2.```bash
  3. 3.# Start memcached with LRU crawler enabled
  4. 4.memcached -o lru_crawler,lru_maintainer

# Or enable on a running instance echo "lru_crawler enable" | nc localhost 11211 echo "lru_crawler crawl all" | nc localhost 11211 ```

  1. 1.**Use modern LRU maintenance":
  2. 2.```bash
  3. 3.# Enable segmented LRU with temp items (Memcached 1.6+)
  4. 4.memcached -o lru_maintainer,lru_crawler,modern

# This creates hot, warm, and cold LRU lists # Hot items are protected from eviction ```

  1. 1.**Increase memory allocation":
  2. 2.```bash
  3. 3.memcached -m 8192 -c 10000 -o lru_maintainer,modern
  4. 4.# -m 8192: 8GB of memory
  5. 5.`
  6. 6.**Set appropriate TTLs to prevent cold data from occupying space":
  7. 7.```python
  8. 8.# Hot configuration: longer TTL
  9. 9.client.set('config:main', config_data, expire=86400) # 24 hours

# Session data: shorter TTL client.set(f'session:{session_id}', session_data, expire=1800) # 30 min

# Expensive computation: medium TTL client.set(f'report:monthly:{month}', report, expire=3600) # 1 hour ```

Prevention - Size Memcached memory to hold the entire working set plus 20% headroom - Enable `lru_maintainer` and `modern` options (Memcached 1.6+) - Use LRU crawler to proactively reclaim expired items - Set appropriate TTLs based on access patterns - Monitor eviction rate and memory utilization continuously - Use separate Memcached instances for different data categories - Implement application-level fallback to regenerate evicted hot keys