Introduction The Memcached LRU crawler is responsible for scanning slab classes and reclaiming memory from expired items. When it is not running, misconfigured, or unable to keep up with the expiration rate, expired items consume memory that could be used for active cache entries, leading to premature evictions of valid data.

Symptoms - `STAT reclaimed` is low or zero despite many items having passed their TTL - `STAT evictions` is high while many items in the cache are expired - Memory usage stays near the limit even though many items should have expired - `STAT curr_items` does not decrease after bulk item expiration - Cache hit rate lower than expected for the traffic pattern

Common Causes - LRU crawler not enabled (not started with `-o lru_crawler`) - Crawler throttled too aggressively by `lru_crawler_sleep` - Slab class with most expired items not being crawled - Crawler disabled after being enabled (`lru_crawler disable`) - Memcached version older than 1.4.23 (when LRU crawler was introduced)

Step-by-Step Fix 1. **Check if LRU crawler is running": ```bash echo "stats" | nc localhost 11211 | grep -E "crawler_|lru_" # If no crawler stats appear, it is not enabled

# Also check memcached process arguments ps aux | grep memcached # Look for: -o lru_crawler ```

  1. 1.**Enable the LRU crawler":
  2. 2.```bash
  3. 3.# Enable the crawler
  4. 4.echo "lru_crawler enable" | nc localhost 11211
  5. 5.# Should return: OK

# Start crawling all slab classes echo "lru_crawler crawl all" | nc localhost 11211 # Should return: OK ```

  1. 1.**Configure crawler aggressiveness":
  2. 2.```bash
  3. 3.# Adjust sleep time between items (microseconds, default 100)
  4. 4.# Lower = more aggressive, higher = less impact on performance
  5. 5.echo "lru_crawler sleep 10" | nc localhost 11211
  6. 6.# Should return: OK

# Check the current setting echo "stats settings" | nc localhost 11211 | grep lru_crawler_sleep ```

  1. 1.**Enable the LRU maintainer for automatic crawling":
  2. 2.```bash
  3. 3.# Start memcached with both options
  4. 4.memcached -m 4096 -o lru_crawler,lru_maintainer

# The maintainer automatically triggers the crawler when needed # and manages the segmented LRU (hot/warm/cold) ```

  1. 1.**Verify memory reclamation":
  2. 2.```bash
  3. 3.# Before and after enabling crawler
  4. 4.echo "stats" | nc localhost 11211 | grep -E "reclaimed|evictions|curr_items"

# After enabling crawler, reclaimed should increase # and evictions should decrease ```

Prevention - Always start Memcached with `-o lru_crawler,lru_maintainer` - Monitor `reclaimed` vs `evictions` ratio with alerting - Set `lru_crawler_sleep` based on the expiration rate of your workload - Use the `modern` option (`-o modern`) for automatic LRU tuning - Test LRU crawler effectiveness in staging with production traffic patterns - Include LRU crawler verification in Memcached deployment checklists - Periodically run `lru_crawler crawl all` during maintenance windows