Your Elasticsearch cluster stopped accepting writes or allocating shards because disk usage crossed the watermark thresholds. This safety mechanism prevents nodes from running completely out of space, but it can catch you off guard during peak operations.

Understanding Disk Watermarks

Elasticsearch uses three disk watermark levels:

  • Low watermark (default 85%): No new shards allocated to this node
  • High watermark (default 90%): Attempts to relocate shards away from this node
  • Flood stage watermark (default 95%): Blocks writes to indices with shards on this node

Error patterns you'll encounter:

bash
the node is above the high watermark, blocking allocation
bash
flood stage watermark exceeded on [node-1], all indices with shards on this node are blocked
bash
disk usage exceeded flood-stage watermark, index [logs-2024-01] is now read-only

Initial Diagnosis

First, identify which nodes have exceeded watermarks and check current disk usage:

```bash # Check disk allocation per node curl -s 'http://localhost:9200/_cat/allocation?v&h=shards,disk.indices,disk.used,disk.avail,disk.total,disk.percent,node'

# Check current watermark settings curl -s 'http://localhost:9200/_cluster/settings?include_defaults=true&flat_settings=true&pretty' | \ jq '.defaults.cluster.routing.allocation.disk'

# Check for blocked indices curl -s 'http://localhost:9200/_cat/indices?v&h=index,health,status,pri,rep' | grep -i "read"

# Find indices in read-only mode curl -s 'http://localhost:9200/*/_settings?pretty' | \ jq 'to_entries[] | select(.value.settings.index.blocks.write == "true") | .key'

# Check cluster allocation explain curl -s 'http://localhost:9200/_cluster/allocation/explain?pretty' | \ jq '.explanation, .node_name_decision' ```

Common Cause 1: Actual Disk Space Shortage

The node genuinely has too much data stored.

Diagnosis:

```bash # Check disk usage at OS level df -h /var/lib/elasticsearch

# For each data node for node in es-node1 es-node2 es-node3; do echo "=== $node ===" ssh $node "df -h /var/lib/elasticsearch && du -sh /var/lib/elasticsearch/*" done

# Check which indices consume most space curl -s 'http://localhost:9200/_cat/indices?v&h=index,store.size,docs.count&s=store.size:desc' | head -20

# Check segment size breakdown curl -s 'http://localhost:9200/_cat/segments?v&h=index,shard,segment,size,size.memory&s=size:desc' | head -20 ```

Solution:

Option 1: Delete unnecessary indices

```bash # Delete old log indices curl -X DELETE "http://localhost:9200/logs-2023-*"

# Delete indices matching a pattern curl -X DELETE "http://localhost:9200/*-old-*"

# Delete specific large indices curl -X DELETE "http://localhost:9200/huge-unused-index" ```

Option 2: Reduce replica count

bash
# Reduce replicas for non-critical indices
curl -X PUT "http://localhost:9200/logs-*/_settings" -H 'Content-Type: application/json' -d'
{
  "number_of_replicas": 0
}'

Option 3: Force merge segments

bash
# Reduce segment count (reduces overhead)
curl -X POST "http://localhost:9200/logs-2024-01/_forcemerge?max_num_segments=1"

Option 4: Clear Elasticsearch caches

```bash # Clear field data cache curl -X POST "http://localhost:9200/_cache/clear?fielddata=true"

# Clear all caches curl -X POST "http://localhost:9200/_cache/clear" ```

Option 5: Add more nodes

bash
# Add a new data node to the cluster
# This requires infrastructure changes but is the proper long-term solution

Common Cause 2: Indices in Read-Only Flood Stage

When flood stage is hit, indices automatically become read-only.

Error pattern: ``json { "index.blocks.write": "true" }

Diagnosis:

```bash # Find blocked indices curl -s 'http://localhost:9200/*/_settings?pretty' | \ jq 'to_entries[] | select(.value.settings.index.blocks.write == "true") | {index: .key, blocks: .value.settings.index.blocks}'

# Check if flood stage watermark triggered curl -s 'http://localhost:9200/_cluster/settings?include_defaults=true&pretty' | \ jq '.defaults.cluster.routing.allocation.disk.watermark.flood_stage' ```

Solution:

After freeing up disk space, unlock the indices:

```bash # First, free disk space (see solutions above)

# Then remove read-only block curl -X PUT "http://localhost:9200/_all/_settings" -H 'Content-Type: application/json' -d' { "index.blocks.read_only_allow_delete": null, "index.blocks.write": null }'

# For specific indices curl -X PUT "http://localhost:9200/logs-2024-01/_settings" -H 'Content-Type: application/json' -d' { "index.blocks.read_only_allow_delete": null, "index.blocks.write": null }'

# Verify writes are working curl -X POST "http://localhost:9200/test-index/_doc" -H 'Content-Type: application/json' -d' { "test": "write test" }' ```

Common Cause 3: Temporary Adjustment of Watermarks

Sometimes you need a temporary fix while working on a long-term solution.

Diagnosis:

bash
# Check current settings
curl -s 'http://localhost:9200/_cluster/settings?pretty'

Solution:

Adjust watermarks temporarily (use with caution):

```bash # Adjust to higher percentages (not recommended for production) curl -X PUT "http://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.disk.watermark.low": "90%", "cluster.routing.allocation.disk.watermark.high": "95%", "cluster.routing.allocation.disk.watermark.flood_stage": "98%" } }'

# Or use absolute byte values (more precise) curl -X PUT "http://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.disk.watermark.low": "50gb", "cluster.routing.allocation.disk.watermark.high": "20gb", "cluster.routing.allocation.disk.watermark.flood_stage": "10gb" } }'

# Return to defaults when done curl -X PUT "http://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.disk.watermark.low": null, "cluster.routing.allocation.disk.watermark.high": null, "cluster.routing.allocation.disk.watermark.flood_stage": null } }' ```

Common Cause 4: Shard Relocation Away from Full Nodes

Proactively move shards from full nodes to healthier ones.

Diagnosis:

```bash # Identify full nodes curl -s 'http://localhost:9200/_cat/allocation?v' | awk '$6 > 85 {print $7}'

# Check shard distribution curl -s 'http://localhost:9200/_cat/shards?v&h=index,shard,prirep,state,node&s=node' ```

Solution:

Exclude full nodes from allocation:

```bash # Exclude specific node from receiving new shards curl -X PUT "http://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.exclude._name": "full-node-1" } }'

# Exclude by IP curl -X PUT "http://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.exclude._ip": "10.0.0.5,10.0.0.6" } }'

# Watch shard relocation watch -n 5 'curl -s http://localhost:9200/_cat/recovery?v&active_only=true'

# After shards moved, remove exclusion curl -X PUT "http://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.exclude._name": null } }' ```

Common Cause 5: ILM Policy Issues

Index Lifecycle Management policies might not be deleting old indices properly.

Diagnosis:

```bash # Check ILM policies curl -s 'http://localhost:9200/_ilm/policy?pretty'

# Check ILM status curl -s 'http://localhost:9200/_ilm/status?pretty'

# Check which indices have ILM applied curl -s 'http://localhost:9200/*/_settings?pretty' | \ jq 'to_entries[] | select(.value.settings.index.lifecycle != null) | {index: .key, policy: .value.settings.index.lifecycle.name}'

# Check ILM execution history curl -s 'http://localhost:9200/_ilm/explain?pretty' ```

Solution:

Fix or update ILM policies:

```bash # Update ILM policy to delete sooner curl -X PUT "http://localhost:9200/_ilm/policy/logs-policy" -H 'Content-Type: application/json' -d' { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_size": "50gb", "max_age": "1d" } } }, "warm": { "min_age": "7d", "actions": { "forcemerge": { "max_num_segments": 1 }, "shrink": { "number_of_shards": 1 } } }, "delete": { "min_age": "30d", "actions": { "delete": {} } } } } }'

# Manually start ILM for stuck indices curl -X POST "http://localhost:9200/_ilm/start" ```

Common Cause 6: Snapshot Repository Taking Space

Snapshot repositories on the same filesystem can consume significant space.

Diagnosis:

```bash # Check snapshot repositories curl -s 'http://localhost:9200/_snapshot?pretty'

# Check snapshot storage usage curl -s 'http://localhost:9200/_snapshot/_status?pretty'

# List snapshots with sizes curl -s 'http://localhost:9200/_cat/snapshots?v&h=id,status,start_time,end_time,repository,shards' ```

Solution:

```bash # Delete old snapshots curl -X DELETE "http://localhost:9200/_snapshot/repo/old-snapshot-1"

# Delete multiple snapshots curl -X DELETE "http://localhost:9200/_snapshot/repo/snap-2023-*"

# Move snapshot repository to different storage curl -X PUT "http://localhost:9200/_snapshot/new-repo" -H 'Content-Type: application/json' -d' { "type": "fs", "settings": { "location": "/mnt/larger-storage/elasticsearch-snapshots" } }' ```

Verification

After resolving disk issues, verify everything is working:

```bash # Check disk allocation status curl -s 'http://localhost:9200/_cat/allocation?v'

# Verify cluster health curl -s 'http://localhost:9200/_cluster/health?pretty'

# Confirm indices are writable curl -X POST "http://localhost:9200/test-write/_doc" -H 'Content-Type: application/json' -d' {"test": true}' curl -X DELETE "http://localhost:9200/test-write"

# Check for any remaining blocks curl -s 'http://localhost:9200/*/_settings?pretty' | \ jq 'to_entries[] | select(.value.settings.index.blocks != null)'

# Verify shard allocation is happening curl -s 'http://localhost:9200/_cat/shards?v' | grep -i "initializing|relocating" ```

Prevention

Set up proactive monitoring:

```yaml groups: - name: elasticsearch_disk rules: - alert: ElasticsearchDiskSpaceWarning expr: elasticsearch_filesystem_data_used_percent > 75 for: 5m labels: severity: warning annotations: summary: "Elasticsearch node {{ $labels.instance }} disk usage at {{ $value }}%"

  • alert: ElasticsearchDiskSpaceCritical
  • expr: elasticsearch_filesystem_data_used_percent > 85
  • for: 1m
  • labels:
  • severity: critical
  • annotations:
  • summary: "Elasticsearch node {{ $labels.instance }} approaching watermark"
  • alert: ElasticsearchFloodStageWatermark
  • expr: elasticsearch_filesystem_data_used_percent > 90
  • for: 1m
  • labels:
  • severity: critical
  • annotations:
  • summary: "Elasticsearch flood stage imminent on {{ $labels.instance }}"
  • `

Implement automated cleanup:

```bash #!/bin/bash # cleanup-old-indices.sh # Run daily via cron

# Delete indices older than 30 days matching pattern OLDER_THAN=$(date -d "-30 days" +%Y.%m.%d)

for index in $(curl -s http://localhost:9200/_cat/indices?h=index | grep "logs-"); do index_date=$(echo $index | grep -oE '[0-9]{4}\.[0-9]{2}\.[0-9]{2}') if [[ "$index_date" < "$OLDER_THAN" ]]; then curl -X DELETE "http://localhost:9200/$index" echo "Deleted $index" fi done ```

Disk watermark issues are Elasticsearch's safety mechanism. Address them by freeing space rather than disabling the safeguards, and implement proper ILM policies to prevent recurrence.