How to Fix Nginx Upstream DNS Resolution

# Nginx Upstream DNS Resolution

Upstream backends defined by hostname work initially but fail when DNS changes. Nginx caches DNS lookups at startup and doesn't refresh them automatically. Dynamic environments with changing IPs - Kubernetes, AWS, container orchestration - require special configuration to handle DNS updates.

Understanding Nginx DNS Caching

1. 1.Nginx resolves hostnames when:
2. 2.Configuration is loaded (startup, reload)
3. 3.resolver directive is configured with variable proxy_pass

Without resolver, DNS is cached forever until reload.

Check current resolution: ```bash # What Nginx sees for upstream sudo nginx -T | grep upstream

# What DNS currently returns dig api.example.com +short ```

If they differ, Nginx has stale DNS.

Common Cause 1: DNS Changes Not Reflected

Backend IP changed, Nginx still uses old IP.

Problematic config: ``nginx location /api { proxy_pass http://api.example.com; }

api.example.com resolved at startup. When DNS changes IP, Nginx connects to wrong address.

Error log: `` connect() failed (111: Connection refused) while connecting to upstream

Solution: Use resolver with variable: ``nginx location /api { resolver 8.8.8.8 8.8.4.4 valid=30s; set $upstream api.example.com; proxy_pass http://$upstream; }

Key elements: - resolver: Specifies DNS servers - valid=30s: Re-resolve after 30 seconds - set $upstream: Variable forces dynamic resolution - proxy_pass http://$upstream: Variable triggers resolution

Common Cause 2: Missing Resolver Directive

Resolver not configured for dynamic resolution.

Problematic config: ``nginx location /api { set $upstream api.example.com; proxy_pass http://$upstream; # Missing resolver - won't work }

Nginx error: `` no resolver defined to resolve api.example.com

Solution: Add resolver: ``nginx location /api { resolver 8.8.8.8 valid=30s; set $upstream api.example.com; proxy_pass http://$upstream; }

Common Cause 3: Resolver Timeout Too Short

DNS resolution times out.

Problematic config: ``nginx resolver 8.8.8.8 valid=30s; # Default timeout is short

Solution: Increase timeout: ``nginx resolver 8.8.8.8 valid=30s ipv6=off; resolver_timeout 10s;

Common Cause 4: Upstream Block with Hostnames

Upstream block also caches DNS.

Problematic config: ```nginx upstream backend { server api1.example.com:8080; server api2.example.com:8080; }

location / { proxy_pass http://backend; } ```

Both hostnames resolved at startup and cached.

Solution: Use resolve parameter (Nginx Plus): ``nginx upstream backend { server api1.example.com:8080 resolve; server api2.example.com:8080 resolve; }

For open source Nginx: ```nginx location / { resolver 8.8.8.8 valid=30s;

# Use multiple backends via variable set $backend1 api1.example.com; set $backend2 api2.example.com;

# This doesn't give you load balancing proxy_pass http://$backend1:8080; } ```

Better approach for open source - use DNS-based LB: ``nginx # Configure DNS to return multiple IPs # Nginx will use all returned IPs resolver 8.8.8.8 valid=30s; set $upstream backend.example.com; proxy_pass http://$upstream:8080;

If DNS returns multiple A records, Nginx uses them all with round-robin.

Common Cause 5: IPv6 Resolution Issues

IPv6 resolution fails or returns addresses Nginx can't use.

Problematic config: ``nginx resolver 8.8.8.8 valid=30s; set $upstream api.example.com; proxy_pass http://$upstream;

If DNS returns IPv6 AAAA records but system can't use IPv6, connection fails.

Solution: Disable IPv6 resolution: ``nginx resolver 8.8.8.8 valid=30s ipv6=off; set $upstream api.example.com; proxy_pass http://$upstream;

Common Cause 6: Internal DNS Server Not Reachable

Resolver points to internal DNS that's not accessible.

Problematic config: ``nginx resolver 10.0.0.100 valid=30s; # Internal DNS might not be reachable from Nginx

Diagnosis: ```bash # Test DNS server from Nginx server dig @10.0.0.100 api.example.com

# Check if DNS port is open nc -zv 10.0.0.100 53 ```

Solution: Verify DNS server accessibility: ```bash # Use accessible DNS resolver 8.8.8.8 valid=30s;

# Or fix internal DNS connectivity ```

Common Cause 7: Kubernetes Service DNS

Kubernetes services change IPs on pod restarts.

Problematic config: ``nginx upstream backend { server backend-service.default.svc.cluster.local:8080; }

Kubernetes service IP might change.

Solution: Use resolver with Kube DNS: ``nginx location / { resolver kube-dns.kube-system.svc.cluster.local valid=30s; set $backend backend-service.default.svc.cluster.local; proxy_pass http://$backend:8080; }

Or use Kubernetes DNS IP directly: ``nginx # Typical Kube DNS is at 10.96.0.10 or similar resolver 10.96.0.10 valid=5s ipv6=off; set $backend backend-service; proxy_pass http://$backend:8080;

Common Cause 8: AWS Route53 Dynamic DNS

AWS services use dynamic DNS that changes.

Solution: ``nginx resolver 169.254.169.253 valid=10s ipv6=off; set $backend my-service.elb.amazonaws.com; proxy_pass http://$backend;

AWS VPC DNS is at 169.254.169.253 (base of VPC IP + 2).

Common Cause 9: Proxy Port Missing

Variable proxy_pass requires port in some cases.

Problematic config: ``nginx resolver 8.8.8.8 valid=30s; set $upstream api.example.com; proxy_pass http://$upstream; # Port 80 by default

If service uses different port:

Solution: Include port: ``nginx resolver 8.8.8.8 valid=30s; set $upstream api.example.com; proxy_pass http://$upstream:8080;

Or encode port in variable: ``nginx set $upstream "api.example.com:8080"; proxy_pass http://$upstream;

Verification Steps

1. 1.Check DNS resolution:
2. 2.```bash
3. 3.dig api.example.com +short
4. 4.`
5. 5.Test resolver configuration:
6. 6.```nginx
7. 7.location /test-resolver {
8. 8.resolver 8.8.8.8 valid=30s;
9. 9.set $backend api.example.com;
10. 10.add_header X-Resolved-IP $upstream_addr always;
11. 11.proxy_pass http://$backend;
12. 12.return 200;
13. 13.}
14. 14.`

curl -I http://localhost/test-resolver | grep X-Resolved-IP

1. 1.Monitor upstream address:
2. 2.```nginx
3. 3.log_format dns '$upstream_addr for $upstream_host [$time_local]';
4. 4.access_log /var/log/nginx/dns.log dns;
5. 5.`
6. 6.Force DNS change and test:
7. 7.```bash
8. 8.# Change DNS record
9. 9.# Wait valid period
10. 10.curl http://localhost/api
11. 11.# Should use new IP
12. 12.`
13. 13.Check resolver status:
14. 14.```nginx
15. 15.resolver 8.8.8.8 valid=30s status_zone=resolver_zone;
16. 16.`

Complete Working Configurations

Dynamic upstream with DNS: ```nginx location /api { resolver 8.8.8.8 8.8.4.4 valid=30s ipv6=off; resolver_timeout 5s;

set $upstream api.example.com; proxy_pass http://$upstream:8080;

proxy_set_header Host $host; proxy_connect_timeout 10s; } ```

Kubernetes configuration: ```nginx resolver 10.96.0.10 valid=5s ipv6=off;

location / { set $backend backend-service.default.svc.cluster.local; proxy_pass http://$backend:8080; } ```

AWS ELB configuration: ```nginx resolver 169.254.169.253 valid=10s ipv6=off;

location / { set $elb my-alb.elb.amazonaws.com; proxy_pass http://$elb; } ```

Multiple backends via DNS round-robin: ```nginx resolver 8.8.8.8 valid=30s;

# DNS must return multiple A records for backend.example.com # Nginx will round-robin between them location / { set $backend backend.example.com; proxy_pass http://$backend; } ```

Quick Reference

Dynamic DNS resolution requires the resolver directive with a variable in proxy_pass. The valid parameter controls refresh timing, and ipv6=off prevents IPv6 resolution issues. For Kubernetes and AWS, use their internal DNS servers.