Introduction Elasticsearch slow query logs capture queries that exceed configured time thresholds. When thresholds are set too low or a poorly optimized query runs frequently, the slow log can grow to gigabytes per day, consuming disk space and potentially causing the node to run out of storage.
Symptoms - `/var/log/elasticsearch/` partition fills up rapidly (GBs per day) - `*_index_search_slowlog.log` or `*_index_indexing_slowlog.log` files are enormous - Disk usage alerts trigger on log partition - Elasticsearch performance degrades due to I/O contention from log writing - `ls -lh /var/log/elasticsearch/` shows multi-gigabyte slow log files
Common Causes - Slow log threshold set too low (e.g., 1ms for search, capturing normal queries) - A single poorly optimized query running thousands of times per minute - No log rotation configured for slow query logs - `index.search.slowlog.threshold.query.warn` set to `0ms` or very low value - Application retry loop executing the same slow query repeatedly
Step-by-Step Fix 1. **Check current slow log thresholds": ```bash curl -s localhost:9200/my_index/_settings | \ jq '.["my_index"].settings.index | to_entries | map(select(.key | contains("slowlog")))' ```
- 1.**Adjust slow log thresholds to reasonable values":
- 2.```bash
- 3.curl -X PUT localhost:9200/my_index/_settings -H 'Content-Type: application/json' -d '{
- 4."index.search.slowlog.threshold.query.warn": "10s",
- 5."index.search.slowlog.threshold.query.info": "5s",
- 6."index.search.slowlog.threshold.query.debug": "2s",
- 7."index.search.slowlog.threshold.query.trace": "500ms",
- 8."index.search.slowlog.threshold.fetch.warn": "5s",
- 9."index.search.slowlog.threshold.fetch.info": "2s",
- 10."index.search.slowlog.threshold.indexing.warn": "10s",
- 11."index.search.slowlog.threshold.indexing.info": "5s"
- 12.}'
- 13.
` - 14.**Disable slow logging temporarily if disk is critical":
- 15.```bash
- 16.curl -X PUT localhost:9200/my_index/_settings -H 'Content-Type: application/json' -d '{
- 17."index.search.slowlog.threshold.query.warn": "-1",
- 18."index.search.slowlog.threshold.query.info": "-1",
- 19."index.search.slowlog.threshold.query.debug": "-1",
- 20."index.search.slowlog.threshold.query.trace": "-1"
- 21.}'
- 22.
` - 23.**Clean up existing large slow log files":
- 24.```bash
- 25.# Find large slow log files
- 26.find /var/log/elasticsearch/ -name "*slowlog*" -size +1G -exec ls -lh {} \;
# Truncate (not delete) the files to free space truncate -s 0 /var/log/elasticsearch/*slowlog.log
# Or compress old logs gzip /var/log/elasticsearch/*slowlog.log.[0-9]* ```
- 1.**Configure log rotation in log4j2.properties":
- 2.
` - 3.# /etc/elasticsearch/log4j2.properties
- 4.appender.index_search_slowlog_rolling.type = RollingFile
- 5.appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling
- 6.appender.index_search_slowlog_rolling.fileName = ${sys:es.logs}_index_search_slowlog.log
- 7.appender.index_search_slowlog_rolling.layout.type = PatternLayout
- 8.appender.index_search_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %m%n
- 9.appender.index_search_slowlog_rolling.filePattern = ${sys:es.logs}_index_search_slowlog-%d{yyyy-MM-dd}.log
- 10.appender.index_search_slowlog_rolling.policies.type = Policies
- 11.appender.index_search_slowlog_rolling.policies.time.type = TimeBasedTriggeringPolicy
- 12.appender.index_search_slowlog_rolling.policies.time.interval = 1
- 13.appender.index_search_slowlog_rolling.policies.time.modulate = true
- 14.appender.index_search_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicy
- 15.appender.index_search_slowlog_rolling.policies.size.size = 100MB
- 16.
`