What's Actually Happening

NATS Streaming server fails to recover state after restart. Messages and subscriptions are lost.

The Error You'll See

```bash $ nats-streaming-server

[ERR] Error recovering state: corruption detected ```

Why This Happens

  1. 1.Data store corruption
  2. 2.Disk space issues
  3. 3.Permission errors
  4. 4.Version mismatch
  5. 5.Improper shutdown

Step 1: Check Logs

bash
journalctl -u nats-streaming -f
tail -f /var/log/nats-streaming.log

Step 2: Check Data Directory

bash
ls -la /var/lib/nats-streaming/
df -h /var/lib/nats-streaming/

Step 3: Check Permissions

bash
ls -la /var/lib/nats-streaming/
chown -R nats:nats /var/lib/nats-streaming/
chmod -R 755 /var/lib/nats-streaming/

Step 4: Verify Store Type

bash
# In config:
store: file
dir: /var/lib/nats-streaming

Step 5: Check Cluster Config

bash
# Verify cluster configuration
cluster:
  node_id: "nats-1"
  peers: ["nats-2", "nats-3"]

Step 6: Clean and Restart

bash
# Backup data first!
cp -r /var/lib/nats-streaming /backup/
rm -rf /var/lib/nats-streaming/*
systemctl restart nats-streaming

Step 7: Check NATS Server

bash
systemctl status nats
nc -zv localhost 4222

Step 8: Verify Recovery

bash
nats-streaming-server -sc /etc/nats-streaming/server.conf

Step 9: Check Disk Space

bash
df -h
# Ensure sufficient space for message store

Step 10: Monitor Server

bash
curl http://localhost:8222/streaming/serverz
watch -n 5 'curl -s http://localhost:8222/streaming/serverz'
  • [Fix NATS JetStream Consumer Pending](/articles/fix-nats-jetstream-consumer-pending-messages)
  • [Fix Kafka Consumer Group Not Rebalancing](/articles/fix-kafka-consumer-group-not-rebalancing)