Fix Kong Gateway 500 Error

What's Actually Happening

Kong API Gateway returns HTTP 500 Internal Server Error for requests. Clients receive error instead of expected response.

The Error You'll See

```bash $ curl http://kong:8000/api/users

HTTP/1.1 500 Internal Server Error { "message": "An unexpected error occurred" } ```

Plugin error:

{
  "message": "plugin 'rate-limiting' failed: rate limit exceeded"
}

Upstream error:

{
  "message": "failure to get a peer from the ring-balancer"
}

Database error:

{
  "message": "database connection failed"
}

Why This Happens

1. 1.Plugin error - Plugin execution fails
2. 2.Upstream unavailable - Target service not responding
3. 3.Database connection - Kong cannot connect to database
4. 4.Configuration error - Invalid service or route config
5. 5.Resource exhaustion - Memory or CPU overloaded
6. 6.SSL/TLS issues - Certificate verification failures

Step 1: Check Kong Status

```bash # Check Kong status: curl http://localhost:8001/status

# Check Kong health: curl http://localhost:8001/health

# Check Kong version: curl http://localhost:8001/

# Check Kong process: ps aux | grep kong

# Check Kong logs: tail -f /usr/local/kong/logs/error.log # Or for Docker: docker logs kong

# Check Kong error log: grep -i error /usr/local/kong/logs/error.log | tail -50

# Check admin API: curl http://localhost:8001

# Kong process info: kong version kong check /etc/kong/kong.conf ```

Step 2: Check Service and Route

```bash # List services: curl http://localhost:8001/services | jq .

# Get specific service: curl http://localhost:8001/services/my-service | jq .

# Check service health: curl http://localhost:8001/services/my-service/health

# List routes: curl http://localhost:8001/routes | jq .

# Check route configuration: curl http://localhost:8001/routes/my-route | jq .

# Verify upstream target: curl http://localhost:8001/upstreams | jq .

# Check upstream targets: curl http://localhost:8001/upstreams/my-upstream/targets | jq .

# Test upstream directly: curl http://upstream-service:8080/health

# Service configuration check: # Verify host, port, path are correct curl http://localhost:8001/services/my-service | jq '{host, port, path}' ```

Step 3: Check Plugins

```bash # List all plugins: curl http://localhost:8001/plugins | jq .

# Get specific plugin: curl http://localhost:8001/plugins/plugin-id | jq .

# Check plugin status: # Look for enabled: true/false

# Common problematic plugins: # - rate-limiting # - request-transformer # - jwt # - oauth2 # - acl

# Disable plugin temporarily: curl -X PATCH http://localhost:8001/plugins/plugin-id \ -d "enabled=false"

# Check plugin configuration: curl http://localhost:8001/plugins/plugin-id | jq '.config'

# Plugin errors in logs: grep -i "plugin" /usr/local/kong/logs/error.log | tail -20

# List global plugins: curl "http://localhost:8001/plugins?enabled=true" | jq '.data[].name' ```

Step 4: Check Upstream Connectivity

```bash # Check upstream targets: curl http://localhost:8001/upstreams/my-upstream/targets | jq .

# Test upstream directly: curl -v http://upstream-host:8080/health

# Check DNS resolution: nslookup upstream-host dig upstream-host

# Check upstream health checks: curl http://localhost:8001/upstreams/my-upstream/health | jq .

# Check active health checks: # In upstream config: # healthchecks.active.healthy.interval # healthchecks.active.unhealthy.interval

# Check passive health checks: # healthchecks.passive.healthy.successes # healthchecks.passive.unhealthy.tcp_failures

# Manually add target: curl -X POST http://localhost:8001/upstreams/my-upstream/targets \ -d "target=upstream-host:8080"

# Remove unhealthy target: curl -X DELETE http://localhost:8001/upstreams/my-upstream/targets/target-id

# Check circuit breaker: # If too many failures, Kong may circuit break # Wait or manually reset ```

Step 5: Check Database Connection

```bash # Kong stores config in database (PostgreSQL or Cassandra)

# Check database config: cat /etc/kong/kong.conf | grep database

# For PostgreSQL: # Check connection: psql -h postgres -U kong -d kong -c "SELECT 1"

# Check database logs: docker logs postgres

# Check Kong database migrations: kong migrations list

# Run migrations if needed: kong migrations up

# Check database connection pool: curl http://localhost:8001/status | jq .database

# For declarative config (DB-less): # Check config file: cat /etc/kong/kong.yml

# Validate declarative config: kong config parse /etc/kong/kong.yml

# Reload Kong: kong reload ```

Step 6: Check Resource Usage

```bash # Check Kong memory: ps aux | grep kong | awk '{print $6}'

# Check system memory: free -m

# Check CPU: top -bn1 | grep kong

# Check Kong workers: curl http://localhost:8001/status | jq .memory.workers

# Check nginx worker processes: ps aux | grep "nginx.*worker" | wc -l

# Kong worker count config: grep worker_processes /etc/kong/nginx.conf

# Adjust worker processes: # In kong.conf: # nginx_worker_processes = auto

# Check open file limits: ulimit -n lsof -p $(pgrep nginx) | wc -l

# Increase limits if needed: ulimit -n 65535 ```

Step 7: Check SSL/TLS Issues

```bash # Check certificate configuration: curl http://localhost:8001/certificates | jq .

# Check snis: curl http://localhost:8001/snis | jq .

# Test SSL: openssl s_client -connect kong:8443 -servername api.example.com

# Check certificate expiration: openssl s_client -connect kong:8443 -servername api.example.com 2>/dev/null | openssl x509 -enddate -noout

# Verify CA bundle: curl http://localhost:8001/certificates/cert-id | jq '.cert'

# Check SSL error in logs: grep -i "ssl|certificate|tls" /usr/local/kong/logs/error.log

# Add certificate: curl -X POST http://localhost:8001/certificates \ -d "cert=@/path/to/cert.pem" \ -d "key=@/path/to/key.pem" \ -d "snis[]=api.example.com"

# Disable SSL verify for upstream (dev only): # In service config: curl -X PATCH http://localhost:8001/services/my-service \ -d "tls_verify=false" ```

Step 8: Check Rate Limiting

```bash # Rate limiting errors often cause 500:

# Check rate-limiting plugin: curl http://localhost:8001/plugins?name=rate-limiting | jq .

# Check rate limit config: curl http://localhost:8001/plugins/plugin-id | jq '.config'

# Common rate limit errors: # - Limit exceeded # - Policy (local/cluster/redis) failing # - Redis connection error

# For Redis rate limiting: # Check Redis connection: redis-cli ping

# Check Redis errors: grep -i redis /usr/local/kong/logs/error.log

# Increase rate limits: curl -X PATCH http://localhost:8001/plugins/plugin-id \ -d "config.second=100" \ -d "config.minute=6000"

# Disable rate limiting temporarily: curl -X PATCH http://localhost:8001/plugins/plugin-id \ -d "enabled=false" ```

Step 9: Enable Debug Logging

```bash # Check current log level: cat /etc/kong/kong.conf | grep log_level

# Set to debug: # In kong.conf: log_level = debug

# Reload Kong: kong reload

# Watch debug logs: tail -f /usr/local/kong/logs/error.log

# Check access logs: tail -f /usr/local/kong/logs/access.log

# Debug specific request: # Add request ID header: curl -H "X-Request-ID: debug-123" http://kong:8000/api

# Find in logs: grep "debug-123" /usr/local/kong/logs/*.log

# Check admin API error details: curl http://localhost:8001/status | jq .

# Kong debug mode: KONG_LOG_LEVEL=debug kong start ```

Step 10: Kong Gateway Verification Script

```bash # Create verification script: cat << 'EOF' > /usr/local/bin/check-kong.sh #!/bin/bash

echo "=== Kong Status ===" curl -s http://localhost:8001/status | jq .

echo "" echo "=== Kong Services ===" curl -s http://localhost:8001/services | jq '.data[] | {name, host, port}'

echo "" echo "=== Kong Routes ===" curl -s http://localhost:8001/routes | jq '.data[] | {name, paths}'

echo "" echo "=== Kong Plugins ===" curl -s http://localhost:8001/plugins | jq '.data[] | {name, enabled}'

echo "" echo "=== Upstreams ===" curl -s http://localhost:8001/upstreams | jq '.data[] | {name}'

echo "" echo "=== Error Log (last 20 lines) ===" tail -20 /usr/local/kong/logs/error.log 2>/dev/null || echo "Cannot read log"

echo "" echo "=== Kong Process ===" ps aux | grep -E "kong|nginx" | grep -v grep

echo "" echo "=== Database Test ===" curl -s http://localhost:8001/status | jq '.database'

echo "" echo "=== Recommendations ===" echo "1. Check Kong error logs for specific error" echo "2. Verify upstream service is healthy" echo "3. Check plugin configurations" echo "4. Verify database connectivity" echo "5. Check rate limiting configuration" echo "6. Verify SSL certificates are valid" echo "7. Check resource usage (memory, CPU)" EOF

chmod +x /usr/local/bin/check-kong.sh

# Usage: /usr/local/bin/check-kong.sh ```

Kong Gateway Checklist

Verify the Fix

```bash # After fixing Kong 500 error

# 1. Check Kong status curl http://localhost:8001/status // All components healthy

# 2. Test service curl http://kong:8000/api/users // Returns 200 OK

# 3. Check error log tail -20 /usr/local/kong/logs/error.log // No recent errors

# 4. Verify upstream health curl http://localhost:8001/upstreams/my-upstream/health // All targets healthy

# 5. Test plugins curl http://localhost:8001/plugins?enabled=true // All enabled plugins working

# 6. Monitor access log tail -f /usr/local/kong/logs/access.log // Requests processed successfully ```

Related Issues

• [Fix Nginx 502 Bad Gateway](/articles/fix-502-bad-gateway)
• [Fix Nginx 504 Gateway Timeout](/articles/fix-504-gateway-timeout)
• [Fix HAProxy Backend Server Down](/articles/fix-haproxy-backend-server-down)