Fix MongoDB Replica Set Initialization Failed

What's Actually Happening

MongoDB replica set fails to initialize. Members cannot elect primary, or rs.initiate() returns errors.

The Error You'll See

```javascript $ rs.initiate()

{ "ok": 0, "errmsg": "quorumCheck failed because not enough voting nodes", "code": 74 } ```

Connection error:

```javascript $ rs.status()

{ "ok": 0, "errmsg": "no replset config has been received", "code": 94 } ```

Member unreachable:

```javascript $ rs.status()

{ "members": [ { "name": "mongo2:27017", "stateStr": "(not reachable/healthy)" } ] } ```

Why This Happens

1. 1.Insufficient members - Not enough voting members for quorum
2. 2.Network connectivity - Members cannot reach each other
3. 3.Hostname mismatch - Wrong hostnames in config
4. 4.Authentication issues - KeyFile mismatch or auth disabled
5. 5.Port blocked - Firewall blocking replication port
6. 6.Already initialized - Replica set already exists

Step 1: Check MongoDB Configuration

```bash # Check mongod config: cat /etc/mongod.conf

# Required for replica set: replication: replSetName: "myReplicaSet"

# Check bindIp: net: bindIp: "0.0.0.0" # Or specific IPs port: 27017

# Check keyFile (if auth enabled): security: keyFile: /etc/mongodb/keyfile

# Start MongoDB: systemctl start mongod

# Check MongoDB status: systemctl status mongod

# Check listening: ss -tlnp | grep 27017 ```

Step 2: Check Member Connectivity

```bash # Test connectivity between members: nc -zv mongo1 27017 nc -zv mongo2 27017 nc -zv mongo3 27017

# Ping members: ping mongo1 ping mongo2 ping mongo3

# Check DNS resolution: dig mongo1 dig mongo2

# Test from MongoDB shell: mongo --host mongo1 --port 27017

# Test connection to each member: mongo mongo1:27017 --eval "db.runCommand({ping:1})" mongo mongo2:27017 --eval "db.runCommand({ping:1})"

# Check firewall: iptables -L -n | grep 27017

# Allow MongoDB port: iptables -I INPUT -p tcp --dport 27017 -j ACCEPT ```

Step 3: Check Hostname Configuration

```javascript // Use correct hostnames in rs.initiate():

// Wrong: rs.initiate({ _id: "myReplicaSet", members: [ {_id: 0, host: "localhost:27017"}, // Wrong - localhost {_id: 1, host: "127.0.0.1:27017"} // Wrong - IP ] })

// Correct - use resolvable hostnames: rs.initiate({ _id: "myReplicaSet", members: [ {_id: 0, host: "mongo1.example.com:27017"}, {_id: 1, host: "mongo2.example.com:27017"}, {_id: 2, host: "mongo3.example.com:27017"} ] })

// Check hostname resolution: // On each member: hostname -f // Should return full hostname

// Ensure /etc/hosts has entries: cat /etc/hosts | grep mongo ```

Step 4: Check Replica Set Quorum

```javascript // Need majority of voting members:

// For 3 members: need 2 votes (2 alive) // For 5 members: need 3 votes (3 alive) // For 7 members: need 4 votes (4 alive)

// Minimum: 3 members recommended // Single node replica set needs special config:

// Single member (not recommended for production): rs.initiate({ _id: "myReplicaSet", members: [{_id: 0, host: "mongo1:27017"}], settings: {protocolVersion: 1} })

// Check voting members: rs.conf().members.forEach(m => print(m._id + " " + m.host + " votes: " + m.votes))

// Arbiter for 2-member set (adds vote without data): rs.initiate({ _id: "myReplicaSet", members: [ {_id: 0, host: "mongo1:27017"}, {_id: 1, host: "mongo2:27017"}, {_id: 2, host: "arbiter:27017", arbiterOnly: true} ] }) ```

Step 5: Check Authentication Configuration

```bash # Check keyFile exists: ls -la /etc/mongodb/keyfile

# KeyFile must be: # - Base64 encoded # - Minimum 6 characters # - Same on all members! # - Owned by mongod user # - Permissions 400 or 600

# Generate keyFile: openssl rand -base64 756 > /etc/mongodb/keyfile chmod 400 /etc/mongodb/keyfile chown mongod:mongod /etc/mongodb/keyfile

# Copy same keyFile to all members! scp /etc/mongodb/keyfile mongo2:/etc/mongodb/keyfile scp /etc/mongodb/keyfile mongo3:/etc/mongodb/keyfile

# Check keyFile content matches: # On each member: md5sum /etc/mongodb/keyfile // Must be identical!

# Check security config: # In mongod.conf: security: keyFile: /etc/mongodb/keyfile authorization: enabled

# If auth disabled, disable keyFile: # security: # authorization: disabled ```

Step 6: Check if Already Initialized

```javascript // Check replica set status: rs.status()

// If already initialized: rs.status().ok === 1

// Check config: rs.conf()

// If exists, don't run rs.initiate() // Instead, add members: rs.add("mongo4:27017")

// Or reconfigure: rs.reconfig({ _id: "myReplicaSet", members: [ {_id: 0, host: "mongo1:27017"}, {_id: 1, host: "mongo2:27017"} ] }, {force: true}) // force only in emergency

// Remove member: rs.remove("mongo3:27017")

// Check if in replica set: db.runCommand({isMaster: 1}) ```

Step 7: Initialize Replica Set Correctly

```javascript // Connect to first member: mongo --host mongo1 --port 27017

// Initialize: rs.initiate({ _id: "myReplicaSet", version: 1, members: [ { _id: 0, host: "mongo1:27017", priority: 2 }, { _id: 1, host: "mongo2:27017", priority: 1 }, { _id: 2, host: "mongo3:27017", priority: 1 } ] })

// Check result: rs.status()

// Wait for primary election: // PRIMARY state shows in status

// Check primary: rs.isMaster().primary ```

Step 8: Check Member State Transitions

```javascript // Check member states: rs.status().members.forEach(m => print(m.name + ": " + m.stateStr))

// States: // STARTUP - initializing // PRIMARY - primary // SECONDARY - secondary // RECOVERING - recovering // UNKNOWN - not reachable // ARBITER - arbiter

// Watch state transitions: // Run repeatedly: rs.status()

// Check election: rs.status().electionCandidate

// Force election (emergency): // Step down current primary: rs.stepDown()

// Or on specific member: mongo --host mongo2 --eval "rs.stepDown()"

// Force member to primary: rs.freeze(120) // Prevent election temporarily rs.stepDown() // Step down current ```

Step 9: Check MongoDB Logs

```bash # Check MongoDB logs: tail -f /var/log/mongodb/mongod.log

# Look for replication errors: grep -i "repl" /var/log/mongodb/mongod.log | grep -i error grep -i "election" /var/log/mongodb/mongod.log grep -i "quorum" /var/log/mongodb/mongod.log

# Check connection errors: grep -i "connection refused" /var/log/mongodb/mongod.log

# Check authentication errors: grep -i "auth" /var/log/mongodb/mongod.log | grep -i error

# Journalctl: journalctl -u mongod -f

# Enable verbose logging: # In mongod.conf: systemLog: verbosity: 1 component: replication: verbosity: 2 ```

Step 10: MongoDB Replica Set Verification Script

```bash # Create verification script: cat << 'EOF' > /usr/local/bin/check-mongo-replicaset.sh #!/bin/bash

echo "=== MongoDB Status ===" systemctl status mongod | head -5

echo "" echo "=== Listening Ports ===" ss -tlnp | grep 27017

echo "" echo "=== Member Connectivity ===" for host in mongo1 mongo2 mongo3; do nc -zv $host 27017 2>&1 done

echo "" echo "=== Replica Set Status ===" mongo --quiet --eval "JSON.stringify(rs.status())" | jq '.members[] | {name, stateStr, uptime}'

echo "" echo "=== Replica Set Config ===" mongo --quiet --eval "JSON.stringify(rs.conf())" | jq '.members[] | {host, priority, votes}'

echo "" echo "=== Primary Check ===" mongo --quiet --eval "rs.isMaster().primary"

echo "" echo "=== KeyFile Check ===" md5sum /etc/mongodb/keyfile 2>/dev/null || echo "No keyFile"

echo "" echo "=== Recent Errors ===" tail -20 /var/log/mongodb/mongod.log | grep -i "error|warn|fail" EOF

chmod +x /usr/local/bin/check-mongo-replicaset.sh

# Run: /usr/local/bin/check-mongo-replicaset.sh

# Monitor: watch -n 5 'mongo --quiet --eval "rs.status().members.forEach(m => print(m.name + \": \" + m.stateStr))"' ```

MongoDB Replica Set Checklist

Verify the Fix

```bash # After fixing replica set

# 1. Check status mongo --eval "rs.status().ok" // 1

# 2. Check primary mongo --eval "rs.isMaster().primary" // mongo1:27017

# 3. Check all members mongo --eval "rs.status().members.forEach(m => print(m.name + ': ' + m.stateStr))" // All SECONDARY or PRIMARY

# 4. Write to primary mongo --host mongo1 --eval "db.test.insert({x:1})" // Write successful

# 5. Read from secondary mongo --host mongo2 --eval "db.test.find()" // Data replicated

# 6. Monitor election mongo --eval "rs.status()" // Stable PRIMARY ```

Related Issues

• [Fix MongoDB Connection Refused](/articles/fix-mongodb-connection-refused)
• [Fix MongoDB Replication Lag High](/articles/fix-mongodb-replication-lag-high)
• [Fix MongoDB Authentication Failed](/articles/fix-mongodb-authentication-failed)