Fix Redis Cluster Slot Assignment Failed

What's Actually Happening

Redis Cluster fails to assign hash slots to nodes. Cluster remains in partially initialized state with some slots unassigned.

The Error You'll See

```bash $ redis-cli --cluster add-node new-node:6379 existing:6379

[ERR] Node new-node:6379 is not empty. Either the node already knows other nodes or contains some data. ```

Slot assignment error:

```bash $ redis-cli cluster addslots 0 1 2

(error) ERR Slot 0 is already busy ```

Cluster check failure:

```bash $ redis-cli --cluster check node:6379

[WARNING] Node node:6379 has slots in migrating state [WARNING] The following slots are not covered: 5000-6000 ```

Cluster state:

```bash $ redis-cli cluster info

cluster_state:fail cluster_slots_assigned:15384 # Should be 16384 ```

Why This Happens

1. 1.Slot already assigned - Hash slot owned by another node
2. 2.Node not empty - New node contains existing data
3. 3.Cluster state inconsistent - Nodes disagree on slot ownership
4. 4.Resharding in progress - Ongoing migration not completed
5. 5.Insufficient masters - Not enough master nodes for slots
6. 6.Network partition - Nodes cannot communicate

Step 1: Check Cluster Status

```bash # Check cluster info: redis-cli cluster info

# Output: # cluster_state:ok # Should be ok # cluster_slots_assigned:16384 # Must be all 16384 # cluster_slots_ok:16384

# Check cluster nodes: redis-cli cluster nodes

# Output format: # node_id ip:port@bus flags master_id link_state slots

# Check cluster health: redis-cli --cluster check redis1:6379

# Check specific node: redis-cli -h node -p 6379 cluster info

# Count assigned slots: redis-cli cluster nodes | grep -o '[[0-9-]*]' | wc -l ```

Step 2: Check Slot Coverage

```bash # Check which slots are assigned: redis-cli cluster slots

# Output: # 1) 1) 0 # Start slot # 2) 1000 # End slot # 3) 1) "10.0.0.1" # Master IP # 2) 6379 # Master port

# Find unassigned slots: redis-cli --cluster check redis1:6379 2>&1 | grep "not covered"

# Check specific slot: redis-cli cluster keyslot mykey # Returns slot number

# Check slot owner: redis-cli cluster nodes | grep "[1234]"

# List all slots by node: redis-cli cluster nodes | awk '{print $1, $NF}' | grep "[" ```

Step 3: Check Node Connectivity

```bash # Check node connectivity: redis-cli -h redis1 ping redis-cli -h redis2 ping redis-cli -h redis3 ping

# Check cluster bus port (port + 10000): nc -zv redis1 16379 nc -zv redis2 16379

# Check all nodes can reach each other: for node in redis1 redis2 redis3; do echo "Testing $node:" redis-cli -h $node cluster nodes | grep -c "connected" done

# Check handshake status: redis-cli cluster nodes | grep handshake

# Meet nodes: redis-cli cluster meet redis2 6379

# Check link state: redis-cli cluster nodes | awk '{print $1, $NF}' # connected or disconnected ```

Step 4: Fix Empty Node Requirement

```bash # For new cluster node, must be empty:

# Check if node has data: redis-cli -h new-node DBSIZE # Should be 0

# Check if node knows other nodes: redis-cli -h new-node cluster nodes # Should show only itself

# Flush existing data: redis-cli -h new-node FLUSHALL

# Reset cluster config: redis-cli -h new-node CLUSTER RESET HARD

# Verify node is empty: redis-cli -h new-node cluster info redis-cli -h new-node DBSIZE

# Now add to cluster: redis-cli --cluster add-node new-node:6379 existing-master:6379 ```

Step 5: Assign Slots to New Master

```bash # For new master without slots:

# Option 1: Reshard from existing master: redis-cli --cluster reshard redis1:6379 \ --cluster-from <source-node-id> \ --cluster-to <target-node-id> \ --cluster-slots 4096 \ --cluster-yes

# Option 2: Interactive reshard: redis-cli --cluster reshard redis1:6379 # Follow prompts: # How many slots? 4096 # Receiving node ID? <target-node-id> # Source node ID? all

# Option 3: Add specific slots manually: redis-cli cluster addslots {0..4095}

# Verify slots assigned: redis-cli cluster nodes | grep <node-id> ```

Step 6: Fix Slot Migration Issues

```bash # Check for migrating/importing slots: redis-cli cluster nodes | grep -E "[|->"

# Output: # [5000-<-node-id] # Importing # [5000->-node-id] # Migrating

# Check specific slot status: redis-cli cluster getkeysinslot 5000 10

# Fix stuck migration:

# 1. Cancel import on target: redis-cli -h target cluster setslot 5000 stable

# 2. Cancel export on source: redis-cli -h source cluster setslot 5000 stable

# 3. Continue migration: redis-cli --cluster reshard redis1:6379

# Or use fix: redis-cli --cluster fix redis1:6379 --cluster-search-multiple-owners ```

Step 7: Add Slots to Empty Master

```bash # If master has no slots:

# Check master node ID: redis-cli cluster nodes | grep master

# Add slots (must be unassigned): redis-cli cluster addslots {0..4095}

# Error if slots already assigned: # (error) ERR Slot 0 is already busy

# Check who owns slots: redis-cli cluster slots | grep "0]"

# Delete slots from wrong owner first: redis-cli -h wrong-owner cluster delslots {0..4095}

# Then assign to correct node: redis-cli -h correct-node cluster addslots {0..4095}

# Verify: redis-cli cluster nodes | grep correct-node-id ```

Step 8: Rebalance Cluster

```bash # Rebalance slots across all masters:

redis-cli --cluster rebalance redis1:6379

# With threshold: redis-cli --cluster rebalance redis1:6379 --cluster-threshold 1

# Simulate first: redis-cli --cluster rebalance redis1:6379 --cluster-simulate

# Use pipeline for speed: redis-cli --cluster rebalance redis1:6379 --cluster-pipeline 100

# Check balance after: redis-cli cluster nodes | awk '/master/{print $NF}' | sort

# Output shows slot ranges per master: # [0-5460] # [5461-10922] # [10923-16383] ```

Step 9: Fix Cluster State Fail

```bash # If cluster_state:fail

# Check why: redis-cli --cluster check redis1:6379

# Common issues: # 1. Slots not covered # 2. Majority of masters not reachable # 3. Slots in migrating state

# Fix slots not covered: redis-cli --cluster fix redis1:6379

# Fix with multiple owners: redis-cli --cluster fix redis1:6379 --cluster-search-multiple-owners

# Force cluster to OK (dangerous): redis-cli cluster setslot 5000 stable # Repeat for all problem slots

# If majority lost (emergency): # Create new cluster from remaining: redis-cli --cluster create redis1:6379 redis2:6379 --cluster-replicas 1 --cluster-force

# This destroys data on failed nodes! ```

Step 10: Redis Cluster Verification Script

```bash # Create verification script: cat << 'EOF' > /usr/local/bin/check-redis-cluster.sh #!/bin/bash

NODE=${1:-"localhost:6379"}

echo "=== Cluster Info ===" redis-cli -h $(echo $NODE | cut -d: -f1) -p $(echo $NODE | cut -d: -f2) cluster info | grep -E "state|slots|known"

echo "" echo "=== Cluster Nodes ===" redis-cli -h $(echo $NODE | cut -d: -f1) -p $(echo $NODE | cut -d: -f2) cluster nodes | awk '{print $2, $3, $8}' | column -t

echo "" echo "=== Slot Coverage ===" redis-cli -h $(echo $NODE | cut -d: -f1) -p $(echo $NODE | cut -d: -f2) cluster slots | wc -l echo "Expected: 16384 slots"

echo "" echo "=== Cluster Check ===" redis-cli --cluster check $NODE 2>&1 | grep -E "[OK]|[ERR]|[WARNING]"

echo "" echo "=== Node Connectivity ===" for node in $(redis-cli -h $(echo $NODE | cut -d: -f1) cluster nodes | awk '{print $2}' | cut -d@ -f1); do host=$(echo $node | cut -d: -f1) port=$(echo $node | cut -d: -f2) redis-cli -h $host -p $port ping done

echo "" echo "=== Unassigned Slots ===" redis-cli --cluster check $NODE 2>&1 | grep "not covered" EOF

chmod +x /usr/local/bin/check-redis-cluster.sh

# Usage: /usr/local/bin/check-redis-cluster.sh redis1:6379

# Monitor: watch -n 5 'redis-cli cluster info | grep cluster_state' ```

Redis Cluster Checklist

Verify the Fix

```bash # After fixing slot assignment

# 1. Check cluster info redis-cli cluster info | grep cluster_state // cluster_state:ok

# 2. Verify slots redis-cli cluster info | grep cluster_slots_assigned // cluster_slots_assigned:16384

# 3. Check nodes redis-cli cluster nodes // All nodes connected, slots assigned

# 4. Test key storage redis-cli set testkey value // OK

# 5. Check key slot redis-cli cluster keyslot testkey // Shows slot number

# 6. Run cluster check redis-cli --cluster check redis1:6379 // All checks pass ```

Related Issues

• [Fix Redis Replication Broken](/articles/fix-redis-replication-broken)
• [Fix Redis Connection Refused](/articles/fix-redis-connection-refused)
• [Fix Redis Memory Out of Memory](/articles/fix-redis-memory-out-of-memory)