Kibana shows "Unable to connect to Elasticsearch" or "Elasticsearch is unreachable" errors, and you cannot access your dashboards or visualizations. This connection issue blocks your entire observability workflow.

Understanding the Error

Connection failures appear in various forms:

In Kibana UI: `` Unable to connect to Elasticsearch

bash
Elasticsearch cluster did not respond

In Kibana logs: `` [error][elasticsearch] Elasticsearch is unreachable: http://elasticsearch:9200

bash
[error][status] Kibana is unable to connect to Elasticsearch

Initial Diagnosis

Start by checking both Kibana and Elasticsearch status:

```bash # Check Kibana status systemctl status kibana

# Check Kibana logs journalctl -u kibana -n 100 --no-pager | grep -i "error|connect|elasticsearch"

# Or check log file tail -n 100 /var/log/kibana/kibana.log | grep -i "error|connect"

# Check Elasticsearch status curl -s http://localhost:9200/_cluster/health?pretty

# Check Kibana configuration cat /etc/kibana/kibana.yml | grep -E "elasticsearch|host|port|url" ```

Common Cause 1: Network Connectivity Issues

The most common cause is that Kibana cannot reach Elasticsearch over the network.

Error pattern: `` Elasticsearch is unreachable: http://elasticsearch:9200

Diagnosis:

```bash # Test Elasticsearch connectivity from Kibana server curl -v http://elasticsearch:9200/_cluster/health

# Check if port is open nc -zv elasticsearch 9200

# Check DNS resolution nslookup elasticsearch dig elasticsearch +short

# For Kubernetes, test from Kibana pod kubectl exec -it kibana-pod -- curl http://elasticsearch:9200/_cluster/health

# Check network routing traceroute elasticsearch ping elasticsearch -c 3 ```

Solution:

Fix network connectivity or update configuration:

```yaml # /etc/kibana/kibana.yml # Option 1: Use correct hostname elasticsearch.hosts: ["http://elasticsearch:9200"]

# Option 2: Use IP address if DNS is broken elasticsearch.hosts: ["http://10.0.0.5:9200"]

# Option 3: Use Kubernetes service name elasticsearch.hosts: ["http://elasticsearch-master.monitoring.svc.cluster.local:9200"]

# Option 4: Multiple hosts for HA elasticsearch.hosts: ["http://es-node1:9200", "http://es-node2:9200", "http://es-node3:9200"] ```

For firewall issues:

```bash # Check firewall rules iptables -L -n | grep 9200

# For firewalld firewall-cmd --list-all

# Allow Elasticsearch traffic firewall-cmd --add-port=9200/tcp --permanent firewall-cmd --reload

# Or allow specific source firewall-cmd --add-rich-rule='rule family="ipv4" source address="kibana-ip" port protocol="tcp" port="9200" accept' --permanent ```

Common Cause 2: Authentication Failures

Elasticsearch requires authentication, but Kibana isn't configured with credentials.

Error pattern: `` [error][elasticsearch] Authentication exception

bash
401 Unauthorized

Diagnosis:

```bash # Test Elasticsearch with credentials curl -u elastic:password http://elasticsearch:9200/_cluster/health

# Check current Kibana auth settings cat /etc/kibana/kibana.yml | grep -i "username|password|auth"

# Check Elasticsearch security settings curl -u elastic:password http://elasticsearch:9200/_xpack/security?pretty

# Test with specific user curl -u kibana_user:password http://elasticsearch:9200/.kibana/_search ```

Solution:

Configure authentication in Kibana:

```yaml # /etc/kibana/kibana.yml elasticsearch.username: "kibana_system" elasticsearch.password: "your_password"

# For Elastic Cloud elasticsearch.username: "elastic" elasticsearch.password: "cloud_password" ```

For better security, use a keystore:

```bash # Create keystore entries /usr/share/kibana/bin/kibana-keystore create /usr/share/kibana/bin/kibana-keystore add elasticsearch.password # Enter password when prompted

# Restart Kibana systemctl restart kibana ```

Create a dedicated Kibana user:

bash
# Create kibana_system user if not exists
curl -u elastic:password -X POST "http://elasticsearch:9200/_security/user/kibana_system" -H 'Content-Type: application/json' -d'
{
  "password": "secure_password",
  "roles": ["kibana_system"],
  "full_name": "Kibana System User"
}'

Common Cause 3: TLS/SSL Certificate Issues

SSL certificate problems prevent secure connections.

Error pattern: `` [error][elasticsearch] Client request error: self signed certificate

bash
[error][elasticsearch] UNABLE_TO_VERIFY_LEAF_SIGNATURE

Diagnosis:

```bash # Check Elasticsearch SSL configuration curl -u elastic:password https://elasticsearch:9200/_cluster/health

# Test with SSL verification disabled curl -k -u elastic:password https://elasticsearch:9200/_cluster/health

# Check certificate details openssl s_client -connect elasticsearch:9200 -servername elasticsearch

# Check certificate expiration echo | openssl s_client -connect elasticsearch:9200 2>/dev/null | openssl x509 -noout -dates ```

Solution:

Configure SSL in Kibana:

```yaml # /etc/kibana/kibana.yml # For SSL connection elasticsearch.hosts: ["https://elasticsearch:9200"]

# Option 1: Skip verification (not recommended for production) elasticsearch.ssl.verificationMode: none

# Option 2: Use CA certificate elasticsearch.ssl.certificateAuthorities: ["/path/to/ca.crt"]

# Option 3: Use system certificates elasticsearch.ssl.certificateAuthorities: ["/etc/pki/tls/certs/ca-bundle.crt"] ```

For Kibana's own SSL:

```yaml # Kibana SSL settings server.ssl.enabled: true server.ssl.certificate: /path/to/kibana.crt server.ssl.key: /path/to/kibana.key

# If using same CA elasticsearch.ssl.certificateAuthorities: ["/path/to/ca.crt"] ```

Common Cause 4: Version Compatibility Issues

Kibana and Elasticsearch versions must match for proper operation.

Error pattern: `` [error][elasticsearch] Version mismatch: Kibana 8.5.0 is not compatible with Elasticsearch 7.17.0

Diagnosis:

```bash # Check Kibana version curl -s http://localhost:5601/api/status | jq '.version'

# Check Elasticsearch version curl -s http://elasticsearch:9200 | jq '.version'

# Check via command line /usr/share/kibana/bin/kibana --version /usr/share/elasticsearch/bin/elasticsearch --version

# Check compatibility matrix # https://www.elastic.co/support/matrix#matrix_compatibility ```

Solution:

Match versions:

```bash # If Kibana is newer, upgrade Elasticsearch # Or downgrade Kibana to match Elasticsearch

# For package installations: yum install kibana-7.17.0 # or apt-get install kibana=7.17.0

# For Docker: docker pull docker.elastic.co/kibana/kibana:7.17.0 docker run -d --name kibana docker.elastic.co/kibana/kibana:7.17.0

# For Kubernetes, update the deployment kubectl set image deployment/kibana kibana=docker.elastic.co/kibana/kibana:7.17.0 ```

Common Cause 5: Elasticsearch Not Running or Unhealthy

Sometimes Elasticsearch itself is the problem.

Error pattern: `` Elasticsearch cluster did not respond

Diagnosis:

```bash # Check Elasticsearch health curl -s http://elasticsearch:9200/_cluster/health?pretty

# Check if Elasticsearch is running systemctl status elasticsearch

# Check Elasticsearch logs journalctl -u elasticsearch -n 50 | grep -i "error|started"

# Check for red cluster status curl -s 'http://elasticsearch:9200/_cat/health?v'

# Check node status curl -s 'http://elasticsearch:9200/_cat/nodes?v' ```

Solution:

Fix Elasticsearch first:

```bash # Start Elasticsearch if stopped systemctl start elasticsearch

# Check for specific Elasticsearch issues # (See fix-elasticsearch-cluster-red article for detailed troubleshooting)

# Restart Elasticsearch systemctl restart elasticsearch

# Wait for cluster to be ready until curl -s http://elasticsearch:9200/_cluster/health | jq '.status' | grep -q "green|yellow"; do echo "Waiting for Elasticsearch..." sleep 5 done

# Then restart Kibana systemctl restart kibana ```

Common Cause 6: Memory and Resource Issues

Kibana may fail to connect due to resource constraints.

Error pattern: `` [error][status] Kibana is unable to connect to Elasticsearch

Diagnosis:

```bash # Check Kibana memory usage ps aux | grep kibana | awk '{print $6}' curl -s http://localhost:5601/api/status | jq '.metrics.process.memory'

# Check system memory free -h

# Check Kibana heap settings ps aux | grep kibana | grep -o "Xmx[0-9]*[gm]"

# Check for memory errors in logs grep -i "out of memory|heap|oom" /var/log/kibana/kibana.log ```

Solution:

Adjust memory settings:

```bash # Add to systemd override systemctl edit kibana

[Service] Environment="NODE_OPTIONS=--max-old-space-size=4096"

# Or in kibana.yml # For Node.js memory limit node.options: "--max-old-space-size=4096"

# Restart Kibana systemctl restart kibana ```

Common Cause 7: Index or Plugin Issues

Kibana requires specific indices and plugins in Elasticsearch.

Error pattern: `` [error][elasticsearch] Index .kibana_1 does not exist

Diagnosis:

```bash # Check Kibana indices curl -s 'http://elasticsearch:9200/_cat/indices?v' | grep kibana

# Check if .kibana index exists curl -s http://elasticsearch:9200/.kibana/_search

# Check Elasticsearch plugins curl -s http://elasticsearch:9200/_cat/plugins?v

# Check for security plugin curl -u elastic:password http://elasticsearch:9200/_xpack ```

Solution:

Create Kibana index if missing:

```bash # Kibana should auto-create the index # If it fails, you may need to migrate

# Check for migration issues curl -s http://localhost:5601/api/status | jq '.status'

# Force migration curl -X POST http://localhost:5601/api/saved_objects/migrate

# Or create index manually curl -u elastic:password -X PUT "http://elasticsearch:9200/.kibana_1" -H 'Content-Type: application/json' -d' { "settings": { "number_of_shards": 1, "number_of_replicas": 1 } }' ```

Common Cause 8: Port and Binding Issues

Kibana might not be listening on the expected port.

Error pattern: `` Cannot connect to Kibana at http://localhost:5601

Diagnosis:

```bash # Check if Kibana is listening netstat -tlnp | grep 5601 ss -tlnp | grep 5601

# Check Kibana server settings cat /etc/kibana/kibana.yml | grep -E "server|port|host"

# Check if port is blocked iptables -L -n | grep 5601

# Test local connection curl -v http://localhost:5601/api/status ```

Solution:

Fix Kibana server configuration:

```yaml # /etc/kibana/kibana.yml server.port: 5601 server.host: "0.0.0.0" # Listen on all interfaces

# Or bind to specific IP server.host: "10.0.0.5" ```

Verification

After fixing, verify the connection:

```bash # Check Kibana status curl -s http://localhost:5601/api/status | jq '.status'

# Test Elasticsearch connection from Kibana curl -s http://localhost:5601/api/status | jq '.dependencies.elasticsearch'

# Check Kibana UI # Open browser and navigate to http://kibana:5601

# Verify data access curl -s 'http://localhost:5601/elasticsearch/.kibana/_search' | jq '.hits.total'

# Check logs for successful connection journalctl -u kibana --since "5 minutes ago" | grep -i "connected|ready|listening" ```

Prevention

Monitor Kibana and Elasticsearch health:

```yaml groups: - name: kibana_health rules: - alert: KibanaUnhealthy expr: kibana_status_status != 1 for: 5m labels: severity: critical annotations: summary: "Kibana is unhealthy"

  • alert: KibanaElasticsearchDisconnect
  • expr: kibana_elasticsearch_connection_status != 1
  • for: 2m
  • labels:
  • severity: critical
  • annotations:
  • summary: "Kibana cannot connect to Elasticsearch"
  • alert: KibanaHighMemory
  • expr: kibana_process_memory_heap_used_bytes / kibana_process_memory_heap_max_bytes > 0.85
  • for: 5m
  • labels:
  • severity: warning
  • annotations:
  • summary: "Kibana heap usage exceeds 85%"
  • `

Kibana-Elasticsearch connection issues usually boil down to network connectivity, authentication, or SSL configuration. Test connectivity from the Kibana server first, then verify credentials and certificates.