Fix ZooKeeper Leader Election Failed

What's Actually Happening

ZooKeeper ensemble cannot elect a leader node. Cluster enters read-only mode or fails to serve requests.

The Error You'll See

```bash $ zkServer.sh status

Error contacting service. It is probably not running. ```

No leader message:

$ zkServer.sh status
Mode: follower  # But no leader exists

Connection timeout:

org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss

Quorum error:

Quorum error: Not enough followers to form quorum

Why This Happens

1. 1.Quorum lost - Too few nodes available
2. 2.Network partition - Nodes cannot communicate
3. 3.Configuration mismatch - Different myid or config
4. 4.Port blocking - Client or election ports blocked
5. 5.Data corruption - Transaction log corrupted
6. 6.Resource exhaustion - Memory or disk issues

Step 1: Check ZooKeeper Status

```bash # Check ZooKeeper process: ps aux | grep zookeeper

# Check service: systemctl status zookeeper

# Check server status: zkServer.sh status

# Check mode (leader/follower): zkServer.sh status | grep Mode

# Check logs: tail -f /var/log/zookeeper/zookeeper.log

# Journalctl: journalctl -u zookeeper -f

# Check port: netstat -tlnp | grep 2181

# ZooKeeper ports: # 2181 - Client port # 2888 - Peer port (followers to leader) # 3888 - Leader election port

# Check configuration: cat /etc/zookeeper/zoo.cfg

# Check myid: cat /var/lib/zookeeper/myid

# Check data directory: ls -la /var/lib/zookeeper/

# Check version: zkServer.sh version ```

Step 2: Check Quorum Status

```bash # ZooKeeper requires majority (N/2 + 1) nodes

# For 3 nodes, need 2 for quorum # For 5 nodes, need 3 for quorum

# Check all nodes: for node in node1 node2 node3; do ssh $node "zkServer.sh status" done

# Check node count in config: grep "server\." /etc/zookeeper/zoo.cfg

# Example config: server.1=node1:2888:3888 server.2=node2:2888:3888 server.3=node3:2888:3888

# Count available nodes: # Must have at least quorum size available

# If 2 of 3 nodes down, quorum lost # Need to restart nodes to regain quorum

# Check leader exists: zkServer.sh status | grep Mode # Should show one leader, rest followers

# Check from client: zkCli.sh -server localhost:2181 stat

# Four-letter commands: echo "stat" | nc localhost 2181 | grep Mode echo "srvr" | nc localhost 2181 | grep Mode echo "mntr" | nc localhost 2181

# Check zk_version: echo "conf" | nc localhost 2181 ```

Step 3: Check Network Connectivity

```bash # Test connectivity between nodes: ping node2 ping node3

# Check client port: nc -zv node2 2181

# Check peer port: nc -zv node2 2888

# Check election port: nc -zv node2 3888

# Check firewall: iptables -L -n | grep -E "2181|2888|3888"

# Allow ZooKeeper ports: iptables -I INPUT -p tcp --dport 2181 -j ACCEPT iptables -I INPUT -p tcp --dport 2888 -j ACCEPT iptables -I INPUT -p tcp --dport 3888 -j ACCEPT

# Using ufw: ufw allow 2181/tcp ufw allow 2888/tcp ufw allow 3888/tcp

# Check DNS resolution: nslookup node2

# Check hostname: hostname

# Check hosts file: cat /etc/hosts | grep zookeeper

# Add to hosts if needed: 192.168.1.1 node1 192.168.1.2 node2 192.168.1.3 node3

# Check network interface: ip addr show

# Check from each node: # All nodes must reach each other ```

Step 4: Verify Configuration

```bash # Check zoo.cfg on all nodes: cat /etc/zookeeper/zoo.cfg

# Compare configurations: for node in node1 node2 node3; do echo "=== $node ===" ssh $node "cat /etc/zookeeper/zoo.cfg" done

# Config must be identical on all nodes # server.X entries must match

# Check data directory: grep dataDir /etc/zookeeper/zoo.cfg dataDir=/var/lib/zookeeper

# Check client port: grep clientPort /etc/zookeeper/zoo.cfg clientPort=2181

# Check tick time: grep tickTime /etc/zookeeper/zoo.cfg tickTime=2000

# Check init and sync limits: grep initLimit /etc/zookeeper/zoo.cfg grep syncLimit /etc/zookeeper/zoo.cfg

# Common values: initLimit=10 syncLimit=5

# Increase if network slow: initLimit=20 syncLimit=10

# Check server entries: grep server /etc/zookeeper/zoo.cfg

# Verify format: server.X=hostname:peerPort:electionPort

# Example: server.1=node1:2888:3888 server.2=node2:2888:3888 server.3=node3:2888:3888 ```

Step 5: Check myid Configuration

```bash # Check myid on each node: cat /var/lib/zookeeper/myid

# myid must match server.X number # Node with server.1 must have myid=1

# Verify myid on all nodes: for node in node1 node2 node3; do echo "=== $node ===" ssh $node "cat /var/lib/zookeeper/myid" done

# Fix myid: echo "1" > /var/lib/zookeeper/myid

# Ensure myid file: ls -la /var/lib/zookeeper/myid

# Common errors: # 1. myid missing # 2. myid doesn't match server.X # 3. Duplicate myid on different nodes

# Create missing myid: echo "2" | sudo tee /var/lib/zookeeper/myid

# Check permissions: chmod 644 /var/lib/zookeeper/myid chown zookeeper:zookeeper /var/lib/zookeeper/myid

# Restart after myid change: systemctl restart zookeeper ```

Step 6: Fix Data Corruption

```bash # Check data directory: ls -la /var/lib/zookeeper/

# Check transaction logs: ls -la /var/lib/zookeeper/version-2/

# Check for corruption: # Look for inconsistent logs

# Check snapshot: ls -la /var/lib/zookeeper/version-2/*.snapshot

# Clean corrupted data: # WARNING: Removes all data

# Stop ZooKeeper: systemctl stop zookeeper

# Clear data: rm -rf /var/lib/zookeeper/version-2/*

# Restart: systemctl start zookeeper

# Or restore from backup: cp /backup/zookeeper/version-2/* /var/lib/zookeeper/version-2/

# Check log size: du -sh /var/lib/zookeeper/version-2/

# Configure autopurge: grep autopurge /etc/zookeeper/zoo.cfg

# Enable autopurge: autopurge.purgeInterval=1 autopurge.snapRetainCount=3

# Manual purge: zkCleanup.sh /var/lib/zookeeper 3

# Check transaction log: # Use org.apache.zookeeper.server.LogFormatter java -cp zookeeper.jar org.apache.zookeeper.server.LogFormatter /var/lib/zookeeper/version-2/log.1 ```

Step 7: Check Resource Issues

```bash # Check memory: free -m

# Check ZooKeeper JVM: ps aux | grep zookeeper | grep -E "Xms|Xmx"

# Check JVM memory config: grep -E "Xms|Xmx" /etc/zookeeper/zookeeper-env.sh

# Set JVM memory: export JVMFLAGS="-Xms1g -Xmx1g"

# Or in environment: echo "JVMFLAGS=\"-Xms2g -Xmx2g\"" >> /etc/zookeeper/zookeeper-env.sh

# Check disk space: df -h /var/lib/zookeeper

# Check disk I/O: iostat -x 1

# Check for OOM: dmesg | grep -i "out of memory" journalctl | grep -i oom

# Check file descriptors: ulimit -n

# Increase file descriptors: ulimit -n 65535

# Or in limits.conf: echo "* soft nofile 65535" >> /etc/security/limits.conf

# Check network buffers: sysctl net.core.rmem_max sysctl net.core.wmem_max

# Increase buffers: sysctl -w net.core.rmem_max=16777216 sysctl -w net.core.wmem_max=16777216 ```

Step 8: Restart Cluster Properly

```bash # Proper restart order:

# 1. Stop all nodes: for node in node1 node2 node3; do ssh $node "systemctl stop zookeeper" done

# 2. Verify all stopped: for node in node1 node2 node3; do ssh $node "ps aux | grep zookeeper" done

# 3. Clear old data if needed: for node in node1 node2 node3; do ssh $node "rm -rf /var/lib/zookeeper/version-2/*" done

# 4. Start nodes one by one: ssh node1 "systemctl start zookeeper" sleep 5 ssh node2 "systemctl start zookeeper" sleep 5 ssh node3 "systemctl start zookeeper"

# 5. Check status: for node in node1 node2 node3; do ssh $node "zkServer.sh status" done

# Wait for leader election: sleep 10

# Check for leader: for node in node1 node2 node3; do echo "=== $node ===" ssh $node "zkServer.sh status | grep Mode" done

# Force restart: zkServer.sh restart

# Start in foreground for debug: zkServer.sh start-foreground ```

Step 9: Debug Leader Election

```bash # Enable debug logging: # In log4j.properties: zookeeper.log.level=DEBUG

# Or environment: export ZOO_LOG4J_PROP="DEBUG,ROLLINGFILE"

# Check logs for election: grep -i "leader|election|quorum" /var/log/zookeeper/zookeeper.log

# Watch election process: tail -f /var/log/zookeeper/zookeeper.log | grep -E "LEADER|FOLLOWER|ELECTION"

# Check election stats: echo "mntr" | nc localhost 2181 | grep -E "leader|synced|followers"

# Check from leader: # On leader node: echo "stat" | nc localhost 2181 | grep Mode # Shows: Mode: leader

# Check follower count: echo "mntr" | nc localhost 2181 | grep zk_followers

# Check synced followers: echo "mntr" | nc localhost 2181 | grep zk_synced_followers

# Check pending syncs: echo "mntr" | nc localhost 2181 | grep zk_pending_syncs

# Monitor latency: echo "mntr" | nc localhost 2181 | grep -i latency

# Check packets received/sent: echo "mntr" | nc localhost 2181 | grep -i packets ```

Step 10: ZooKeeper Verification Script

```bash # Create verification script: cat << 'EOF' > /usr/local/bin/check-zookeeper.sh #!/bin/bash

echo "=== ZooKeeper Process ===" ps aux | grep zookeeper | grep -v grep || echo "No ZooKeeper process"

echo "" echo "=== Service Status ===" systemctl status zookeeper 2>/dev/null || echo "Service not running"

echo "" echo "=== Server Status ===" zkServer.sh status 2>&1 || echo "Cannot get status"

echo "" echo "=== Mode ===" echo "stat" | nc localhost 2181 2>/dev/null | grep Mode || echo "Cannot determine mode"

echo "" echo "=== Configuration ===" cat /etc/zookeeper/zoo.cfg 2>/dev/null | head -20 || echo "No config file"

echo "" echo "=== Server Entries ===" grep "server\." /etc/zookeeper/zoo.cfg 2>/dev/null || echo "No server entries"

echo "" echo "=== myid ===" cat /var/lib/zookeeper/myid 2>/dev/null || echo "No myid file"

echo "" echo "=== Data Directory ===" ls -la /var/lib/zookeeper/ 2>/dev/null || echo "No data directory"

echo "" echo "=== Transaction Logs ===" ls -la /var/lib/zookeeper/version-2/ 2>/dev/null | head -10 || echo "No transaction logs"

echo "" echo "=== Listening Ports ===" netstat -tlnp 2>/dev/null | grep java || ss -tlnp | grep java

echo "" echo "=== Monitor Stats ===" echo "mntr" | nc localhost 2181 2>/dev/null | head -20 || echo "Cannot connect"

echo "" echo "=== Network Connectivity ===" for node in $(grep "server\." /etc/zookeeper/zoo.cfg 2>/dev/null | cut -d= -f2 | cut -d: -f1); do echo "Node: $node" ping -c 2 -W 2 $node 2>&1 | tail -2 done

echo "" echo "=== Quorum Check ===" nodes=$(grep "server\." /etc/zookeeper/zoo.cfg 2>/dev/null | wc -l) required=$((nodes / 2 + 1)) echo "Configured nodes: $nodes, Required for quorum: $required"

echo "" echo "=== Recommendations ===" echo "1. Ensure quorum size nodes are available" echo "2. Check all nodes can reach each other" echo "3. Verify myid matches server.X number" echo "4. Allow ports 2181, 2888, 3888 in firewall" echo "5. Compare config across all nodes" echo "6. Check data directory permissions" echo "7. Restart cluster if leader election stuck" EOF

chmod +x /usr/local/bin/check-zookeeper.sh

# Usage: /usr/local/bin/check-zookeeper.sh ```

ZooKeeper Leader Election Checklist

Verify the Fix

```bash # After fixing ZooKeeper leader election

# 1. Check server status zkServer.sh status // Mode: leader or follower

# 2. Check all nodes for node in node1 node2 node3; do ssh $node "zkServer.sh status"; done // One leader, rest followers

# 3. Monitor stats echo "mntr" | nc localhost 2181 // Shows cluster stats

# 4. Test client connection zkCli.sh -server localhost:2181 // Connected to cluster

# 5. Create znode create /test "data" // Znode created

# 6. Check logs tail -f /var/log/zookeeper/zookeeper.log // No election errors ```

Related Issues

• [Fix Kafka Broker Not Starting](/articles/fix-kafka-broker-not-starting)
• [Fix Hadoop NameNode Not Starting](/articles/fix-hadoop-namenode-not-starting)
• [Fix etcd Cluster Unhealthy](/articles/fix-etcd-cluster-unhealthy)