Introduction Cassandra schema disagreement occurs when nodes have different versions of the keyspace/table schema. This commonly happens during rolling upgrades, network partitions, or when schema changes are applied while some nodes are unreachable. Nodes with stale schemas may return incorrect results or reject queries entirely.

Symptoms - `Schema disagreement` warnings in Cassandra logs - `nodetool describecluster` shows `Schema versions` with multiple entries - Some nodes return errors for queries that work on other nodes - Application sees inconsistent behavior depending on which coordinator it connects to - `DESCRIBE KEYSPACE` returns different definitions on different nodes

Common Causes - Rolling upgrade where some nodes are on the old version - Network partition preventing schema gossip propagation - Schema change applied while a node was down - Schema merge failure due to conflicting concurrent schema changes - Old schema versions not being garbage collected

Step-by-Step Fix 1. **Check schema versions across the cluster": ```bash nodetool describecluster # Look for multiple schema versions: # Schema versions: # abc123-...: [node1, node2, node3] # def456-...: [node4] <-- Disagreement! ```

  1. 1.**Force schema pull on the out-of-sync node":
  2. 2.```bash
  3. 3.# On the node with the stale schema
  4. 4.nodetool reloadtriggers
  5. 5.nodetool describecluster # Verify schema version matches

# Or use nodetool resetlocalschema (Cassandra 3.0+) nodetool resetlocalschema ```

  1. 1.**If schema pull fails, manually apply the schema":
  2. 2.```bash
  3. 3.# Get the correct schema from a healthy node
  4. 4.cqlsh healthy-node -e "DESCRIBE KEYSPACE mykeyspace;" > schema.cql

# Apply on the out-of-sync node cqlsh out-of-sync-node -f schema.cql ```

  1. 1.**Restart the affected node as a last resort":
  2. 2.```bash
  3. 3.sudo systemctl restart cassandra

# After restart, verify schema nodetool describecluster ```

  1. 1.**Prevent future disagreements during upgrades":
  2. 2.```bash
  3. 3.# During rolling upgrade:
  4. 4.# 1. Drain the node before upgrading
  5. 5.nodetool drain

# 2. Upgrade and restart sudo systemctl stop cassandra # Install new version sudo systemctl start cassandra

# 3. Verify schema matches before moving to the next node nodetool describecluster ```

Prevention - Always run `nodetool describecluster` after schema changes to verify agreement - Wait for schema agreement before making the next schema change - Use `nodetool enable/disable gossip` during maintenance to prevent partial schema propagation - Apply schema changes from a single, well-connected node - Monitor schema versions with automated checks - Test rolling upgrade procedures in staging with the full schema - Use configuration management to track expected schema versions per node