Redis Persistence Failed

# Redis Persistence Failed

Error Messages

bash

Background saving error: Failed opening the RDB file for saving: Permission denied

Or:

bash

MISCONF Redis is configured to save RDB snapshots, but it's currently not able to persist on disk

Or:

bash

Opening the temp file for AOF rewrite: No space left on device

Or:

bash

AOF rewrite: Write error writing append only file on disk: No space left on device

Root Causes

1.Disk space insufficient - Not enough space for persistence files
2.Permission issues - Redis can't write to data directory
3.Memory issues - Not enough memory for fork() during save
4.Corrupted AOF file - AOF file has invalid commands
5.I/O errors - Disk or filesystem issues
6.Fork failure - Cannot fork process for background save

Diagnosis Steps

Step 1: Check Redis Logs

```bash # Find log location redis-cli CONFIG GET logfile

# View logs tail -100 /var/log/redis/redis-server.log

# Or check systemd journal journalctl -u redis-server -n 100 ```

Step 2: Check Disk Space

```bash # Check disk usage df -h /var/lib/redis

# Check inode usage df -i /var/lib/redis ```

Step 3: Check Directory Permissions

```bash # Check Redis data directory ls -la /var/lib/redis

# Check Redis user ps aux | grep redis

# Check ownership stat /var/lib/redis ```

Step 4: Check Persistence Configuration

```bash # Check RDB settings redis-cli CONFIG GET save

# Check AOF settings redis-cli CONFIG GET appendonly redis-cli CONFIG GET appendfilename

# Check data directory redis-cli CONFIG GET dir ```

Step 5: Check Memory for Fork

```bash # Check available memory free -h

# Check overcommit settings cat /proc/sys/vm/overcommit_memory ```

Solutions

Solution 1: Fix Disk Space Issues

```bash # Check disk usage df -h /var/lib/redis

# Find large files du -sh /var/lib/redis/* | sort -rh

# Clean up old backups rm /var/lib/redis/dump.rdb.old

# Clean up AOF temp files rm /var/lib/redis/temp-rewriteaof-*.aof

# If disk is full on other partitions, clean up sudo apt-get clean # Ubuntu sudo yum clean all # CentOS ```

Solution 2: Fix Permission Issues

```bash # Fix directory ownership sudo chown -R redis:redis /var/lib/redis

# Fix permissions sudo chmod 750 /var/lib/redis

# Ensure Redis can write sudo -u redis touch /var/lib/redis/test sudo -u redis rm /var/lib/redis/test ```

Solution 3: Configure Memory for Fork

```bash # Enable overcommit (recommended for Redis) sudo sysctl vm.overcommit_memory=1

# Make permanent echo "vm.overcommit_memory=1" | sudo tee -a /etc/sysctl.conf

# Disable THP (Transparent Huge Pages) sudo echo never > /sys/kernel/mm/transparent_hugepage/enabled

# Make permanent in /etc/rc.local echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' | sudo tee -a /etc/rc.local ```

Solution 4: Fix RDB Persistence

```bash # Check current RDB status redis-cli LASTSAVE redis-cli INFO persistence

# Trigger manual save redis-cli BGSAVE

# Check if save is working redis-cli LASTSAVE

# If save fails, check error redis-cli INFO persistence | grep rdb_last_bgsave_status

# Disable RDB if not needed redis-cli CONFIG SET save ""

# Or configure save intervals redis-cli CONFIG SET save "900 1 300 10 60 10000" ```

Solution 5: Fix AOF Persistence

```bash # Check AOF status redis-cli INFO persistence

# If AOF is corrupted, use redis-check-aof redis-check-aof /var/lib/redis/appendonly.aof

# Fix corrupted AOF redis-check-aof --fix /var/lib/redis/appendonly.aof

# Manual AOF rewrite redis-cli BGREWRITEAOF

# Enable AOF if disabled redis-cli CONFIG SET appendonly yes redis-cli CONFIG SET appendfilename "appendonly.aof" ```

Solution 6: Repair Corrupted AOF File

```bash # Stop Redis sudo systemctl stop redis-server

# Backup AOF file cp /var/lib/redis/appendonly.aof /var/lib/redis/appendonly.aof.backup

# Check and fix redis-check-aof --fix /var/lib/redis/appendonly.aof

# When prompted, choose to truncate the file # This removes corrupted commands at the end

# Start Redis sudo systemctl start redis-server ```

Solution 7: Handle Low Memory Situations

```bash # Reduce memory usage redis-cli CONFIG SET maxmemory 2gb redis-cli CONFIG SET maxmemory-policy allkeys-lru

# Free memory by deleting unused keys redis-cli --scan --pattern "temp:*" | head -1000 | xargs redis-cli DEL

# Reduce save frequency redis-cli CONFIG SET save "900 100" ```

Solution 8: Change Persistence Strategy

If persistence is causing issues, consider hybrid approach:

```bash # Configure both RDB and AOF redis-cli CONFIG SET save "900 1 300 10 60 10000" redis-cli CONFIG SET appendonly yes redis-cli CONFIG SET appendfsync everysec

# Or use RDB only (faster, less reliable) redis-cli CONFIG SET save "900 1 300 10" redis-cli CONFIG SET appendonly no

# Or AOF only (slower, more reliable) redis-cli CONFIG SET save "" redis-cli CONFIG SET appendonly yes redis-cli CONFIG SET appendfsync everysec ```

Configuration Best Practices

```ini # /etc/redis/redis.conf

# RDB Persistence save 900 1 # Save after 900 seconds if at least 1 key changed save 300 10 # Save after 300 seconds if at least 10 keys changed save 60 10000 # Save after 60 seconds if at least 10000 keys changed

# RDB file name dbfilename dump.rdb

# AOF Persistence appendonly yes appendfilename "appendonly.aof" appendfsync everysec # Sync every second (balanced)

# AOF rewrite settings auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb

# Directory dir /var/lib/redis

# Stop writes on save error (set to no if you want writes to continue) stop-writes-on-bgsave-error yes ```

Monitoring and Alerts

```bash #!/bin/bash # redis_persistence_check.sh

# Check last save LASTSAVE=$(redis-cli LASTSAVE) NOW=$(date +%s) DIFF=$((NOW - LASTSAVE))

# Alert if haven't saved in 1 hour if [ $DIFF -gt 3600 ]; then echo "WARNING: Redis hasn't saved in $DIFF seconds" # Send alert fi

if [ "$RDB_STATUS" != "ok" ]; then echo "ERROR: RDB persistence failed" fi

if [ "$AOF_STATUS" != "ok" ]; then echo "ERROR: AOF persistence failed" fi

# Check disk space USAGE=$(df -h /var/lib/redis | tail -1 | awk '{print $5}' | sed 's/%//') if [ $USAGE -gt 80 ]; then echo "WARNING: Disk usage at ${USAGE}%" fi ```

Recovery Procedures

Recover from RDB Backup

```bash # Stop Redis sudo systemctl stop redis-server

# Restore RDB file cp /backup/dump.rdb /var/lib/redis/dump.rdb chown redis:redis /var/lib/redis/dump.rdb

# Start Redis sudo systemctl start redis-server ```

Recover from AOF Backup

```bash # Stop Redis sudo systemctl stop redis-server

# Restore AOF file cp /backup/appendonly.aof /var/lib/redis/appendonly.aof chown redis:redis /var/lib/redis/appendonly.aof

# If AOF is corrupted, fix it redis-check-aof --fix /var/lib/redis/appendonly.aof

# Start Redis sudo systemctl start redis-server ```

Prevention

1. Regular Backups

```bash #!/bin/bash # redis_backup.sh

BACKUP_DIR="/backup/redis" DATE=$(date +%Y%m%d_%H%M%S)

# Create backup directory mkdir -p $BACKUP_DIR

# Trigger save redis-cli BGSAVE

# Wait for save to complete sleep 5

# Copy RDB file cp /var/lib/redis/dump.rdb $BACKUP_DIR/dump_$DATE.rdb

# Keep only last 7 days find $BACKUP_DIR -name "dump_*.rdb" -mtime +7 -delete ```

2. Monitor Persistence Status

bash

# Add to monitoring
redis-cli INFO persistence | grep -E "rdb_last_bgsave_status|aof_last_bgrewrite_status"

3. Test Restores Regularly

bash

# Monthly restore test
redis-cli --rdb /tmp/test_dump.rdb
# Verify file is valid

[Redis Out of Memory](./fix-redis-out-of-memory)
[Redis Disk Full](./fix-redis-out-of-memory)

Error Messages

Root Causes

Diagnosis Steps

Step 1: Check Redis Logs

Step 2: Check Disk Space

Step 3: Check Directory Permissions

Step 4: Check Persistence Configuration

Step 5: Check Memory for Fork

Solutions

Solution 1: Fix Disk Space Issues

Solution 2: Fix Permission Issues

Solution 3: Configure Memory for Fork

Solution 4: Fix RDB Persistence

Solution 5: Fix AOF Persistence

Solution 6: Repair Corrupted AOF File

Solution 7: Handle Low Memory Situations

Solution 8: Change Persistence Strategy

Configuration Best Practices

Monitoring and Alerts

Recovery Procedures

Recover from RDB Backup

Recover from AOF Backup

Prevention

1. Regular Backups

2. Monitor Persistence Status

3. Test Restores Regularly

Related Errors

Share this guide

More Redis Troubleshooting Guides

Redis Persistence Disabled Warning

Redis Client Output Buffer Exceeded

Redis Slow Log Not Logging

Redis AOF Load Error

Redis Loading RDB Error

Redis TLS Handshake Failed