What's Actually Happening

Cassandra cluster nodes show as down or unreachable. The cluster loses availability or consistency.

The Error You'll See

```bash $ nodetool status

Why This Happens

  1. 1.Network connectivity issues
  2. 2.Gossip protocol problems
  3. 3.Disk space exhausted
  4. 4.Memory issues
  5. 5.Configuration errors

Step 1: Check Node Status

bash
nodetool status
nodetool netstats
nodetool gossipinfo

Step 2: Check Cassandra Logs

bash
tail -f /var/log/cassandra/system.log
grep -i "error\|exception" /var/log/cassandra/system.log

Step 3: Check Network Connectivity

bash
# Check node-to-node connectivity
ping -c 3 other-node
nc -zv other-node 7000
nc -zv other-node 9042

Step 4: Check Disk Space

bash
df -h /var/lib/cassandra
nodetool cfstats | grep "Space used"

Step 5: Check Memory

bash
free -m
cat /proc/meminfo | grep -i mem

Step 6: Check Gossip

bash
nodetool gossipinfo
nodetool disablegossip
nodetool enablegossip

Step 7: Restart Cassandra

bash
systemctl restart cassandra
nodetool drain
systemctl stop cassandra
systemctl start cassandra

Step 8: Decommission Failed Node

bash
nodetool decommission
nodetool removenode <host-id>

Step 9: Check Seed Nodes

yaml
# In cassandra.yaml
seed_provider:
  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
    parameters:
      - seeds: "10.0.0.1,10.0.0.2"

Step 10: Repair Cluster

bash
nodetool repair
nodetool repair -pr
  • [Fix Cassandra Write Timeout](/articles/fix-cassandra-write-timeout)
  • [Fix Cassandra Read Timeout](/articles/fix-cassandra-read-timeout)