What's Actually Happening
Services registered in Consul are not being discovered by clients. Applications cannot find services, DNS queries return no results, or API queries show empty service lists.
The Error You'll See
DNS query empty:
```bash dig @localhost -p 8600 myservice.service.consul
NXDOMAIN ```
API query empty:
```bash curl -s http://localhost:8500/v1/catalog/service/myservice | jq
[] ```
Why This Happens
- 1.Service not registered - Application did not register with Consul
- 2.Health check failing - Service health check returns unhealthy
- 3.Agent disconnected - Consul agent not connected to server
- 4.Wrong service name - Incorrect service name in query
- 5.ACL blocking - ACL policy denies access
- 6.Datacenter mismatch - Service in different datacenter
- 7.Network partition - Network split between client and server
Step 1: Check Consul Agent Status
```bash systemctl status consul
ps aux | grep consul
ss -tlnp | grep 8300 ss -tlnp | grep 8500 ss -tlnp | grep 8600
consul members
journalctl -u consul -f
consul info | grep server ```
Step 2: Check Service Registration
```bash curl -s http://localhost:8500/v1/catalog/services | jq
curl -s http://localhost:8500/v1/catalog/service/myservice | jq
dig @localhost -p 8600 myservice.service.consul +short
curl -s http://localhost:8500/v1/health/service/myservice | jq ```
Step 3: Check Health Check Status
```bash curl -s http://localhost:8500/v1/agent/checks | jq
curl -s http://localhost:8500/v1/health/service/myservice | jq
curl -s http://localhost:8500/v1/health/state/critical | jq ```
Step 4: Check Agent Configuration
```bash cat /etc/consul/consul.json
ls /etc/consul/*.json
grep server /etc/consul/consul.json
grep datacenter /etc/consul/consul.json
consul validate /etc/consul/consul.json
systemctl restart consul ```
Step 5: Check Service Registration Configuration
```bash cat /etc/consul/services/*.json
ls /etc/consul/*.json | xargs jq .name
consul reload
curl -X PUT http://localhost:8500/v1/agent/reload ```
Step 6: Check Consul DNS
```bash dig @localhost -p 8600 consul.service.consul +short
dig @localhost -p 8600 myservice.service.consul +short
dig @localhost -p 8600 myservice.service.consul SRV
grep recursors /etc/consul/consul.json ```
Step 7: Check ACL Configuration
```bash grep acl /etc/consul/consul.json
curl -s -H "X-Consul-Token: master-token" http://localhost:8500/v1/acl/list | jq
curl -s -H "X-Consul-Token: token" http://localhost:8500/v1/agent/services | jq ```
Step 8: Check Datacenter Configuration
```bash curl -s http://localhost:8500/v1/catalog/datacenters | jq
consul info | grep Datacenter
curl -s http://localhost:8500/v1/catalog/service/myservice?dc=dc2 | jq
consul members -wan ```
Step 9: Check Network Connectivity
```bash ping -c 3 consul-server
nc -zv consul-server 8500 nc -zv consul-server 8600 nc -zv consul-server 8300
iptables -L -n | grep 8500 ```
Step 10: Monitor Consul Services
```bash curl -s http://localhost:8500/v1/catalog/services | jq length
curl -s http://localhost:8500/v1/health/state/any | jq
watch -n 5 curl -s http://localhost:8500/v1/catalog/services | jq keys
journalctl -u consul -f
curl -s http://localhost:8500/v1/agent/metrics | jq ```
Consul Service Discovery Checklist
| Check | Command | Expected |
|---|---|---|
| Agent status | consul members | Alive |
| Service registered | /v1/catalog/service | Service listed |
| Health check | /v1/health/service | passing |
| DNS query | dig @localhost:8600 | IP returned |
Verify the Fix
```bash curl -s http://localhost:8500/v1/catalog/service/myservice | jq length
curl -s http://localhost:8500/v1/health/service/myservice | jq .[].Checks[].Status
dig @localhost -p 8600 myservice.service.consul +short
journalctl -u consul --since "5 min ago" | grep -i error ```
Related Issues
- [Fix Consul Agent Not Starting](/articles/fix-consul-agent-not-starting)
- [Fix Prometheus Scrape Error](/articles/fix-prometheus-scrape-error)