Loki is rejecting logs with "rate limit exceeded" errors, causing gaps in your log data. This happens when log ingestion exceeds configured limits, and you're losing visibility into application behavior. Let's diagnose and fix ingestion rate limit issues.

Understanding Rate Limiting

Loki enforces several types of rate limits:

  • Global ingestion rate limit: Overall throughput limit
  • Per-stream rate limit: Limit per log stream (unique label combination)
  • Per-tenant limit: Limits for specific tenants (multi-tenant mode)
  • Stream count limit: Maximum number of unique streams

Error patterns:

bash
ingestion rate limit exceeded (stream: label_set)
bash
per-stream rate limit exceeded: maximum 3MB per stream
bash
tenant ingestion limit exceeded
bash
too many streams: maximum stream count exceeded

Initial Diagnosis

Check Loki logs and ingestion metrics:

```bash # Check Loki logs for rate limit errors kubectl logs -l app=loki -n monitoring | grep -i "rate limit|exceeded" # Or for direct installation journalctl -u loki | grep -i "rate limit"

# Check Loki metrics endpoint curl -s http://localhost:3100/metrics | grep -E "loki_ingester|loki_distributor"

# Get current limits from Loki config curl -s http://localhost:3100/config | jq '.limits'

# Check ingestion rate metrics curl -s http://localhost:3100/metrics | grep loki_ingester_chunks_written_total

# Check rejected logs count curl -s http://localhost:3100/metrics | grep -E "loki_distributor_lines|rejected"

# Monitor real-time ingestion curl -s http://localhost:3100/ready ```

Common Cause 1: Per-Stream Rate Limit Exceeded

A single log stream (unique label combination) exceeds its rate limit.

Error pattern: `` per-stream rate limit exceeded (stream: {app="nginx",namespace="production"})

Diagnosis:

```bash # Check current per-stream limits curl -s http://localhost:3100/config | jq '.limits.per_stream_rate_limit'

# Find high-volume streams curl -s http://localhost:3100/metrics | grep loki_ingester_streams_created_total

# Query Loki for stream cardinality curl -s 'http://localhost:3100/loki/api/v1/labels' | jq '.data'

# Check specific stream volume logcli query '{app="nginx"}' --limit=1000 --since=1h

# Get stream counts curl -s http://localhost:3100/metrics | \ grep -E "loki_ingester_streams|loki_chunk_store"

# Check which labels create most streams curl -s 'http://localhost:3100/loki/api/v1/label/app/values' | jq '.data' ```

Solution:

Increase per-stream limit:

```yaml # loki-config.yaml limits_config: per_stream_rate_limit: 10MB # Increase from default (3MB) per_stream_rate_limit_burst: 20MB # Allow burst ingestion_rate_mb: 20 # Overall ingestion rate ingestion_burst_size_mb: 30

# For Docker/Kubernetes deployment # Update via config map or environment kubectl edit configmap loki-config -n monitoring ```

Or reduce stream volume by limiting labels:

```yaml # In Promtail/Prometheus agent configuration # Reduce labels that create unique streams clients: - url: http://loki:3100/loki/api/v1/push

scrape_configs: - job_name: nginx pipeline_stages: - labels: app: nginx level: level # Only keep necessary labels # Drop labels that cause cardinality - match: selector: '{app="nginx"}' stages: - drop_labels: labels: - pod_name - container_id - stream ```

Common Cause 2: Global Ingestion Rate Limit

Overall ingestion exceeds the global limit.

Error pattern: `` ingestion rate limit exceeded: global limit reached

Diagnosis:

```bash # Check global ingestion limits curl -s http://localhost:3100/config | jq '.limits.ingestion_rate_mb'

# Monitor current ingestion rate curl -s http://localhost:3100/metrics | grep -E "loki_distributor_bytes_received_total" curl -s http://localhost:3100/metrics | grep "rate(loki_distributor_bytes_received_total[5m])"

# Calculate ingestion rate # Use Loki metrics to estimate curl -s 'http://localhost:3100/metrics' | \ grep loki_ingester_chunks_written_total | \ awk '{sum += $2} END {print "Total chunks: " sum}'

# Check burst size curl -s http://localhost:3100/config | jq '.limits.ingestion_burst_size_mb'

# Compare configured vs actual rate ACTUAL=$(curl -s http://localhost:3100/metrics | \ grep "rate(loki_distributor_bytes_received_total[5m])" | \ awk '{print $2}') CONFIG=$(curl -s http://localhost:3100/config | jq '.limits.ingestion_rate_mb') echo "Actual: $ACTUAL MB/s, Configured: $CONFIG MB/s" ```

Solution:

Increase global ingestion limits:

```yaml # loki-config.yaml limits_config: ingestion_rate_mb: 30 # Increase from default ingestion_burst_size_mb: 50 # Allow burst size max_global_streams_per_user: 50000 # More streams allowed

# For high-volume deployments limits_config: ingestion_rate_mb: 100 ingestion_burst_size_mb: 150 per_stream_rate_limit: 20MB per_stream_rate_limit_burst: 40MB

# Apply new configuration kubectl rollout restart deployment/loki -n monitoring ```

Optimize log collection:

yaml
# Reduce log volume with Promtail filtering
scrape_configs:
  - job_name: apps
    pipeline_stages:
      # Drop debug logs to reduce volume
      - match:
          selector: '{level="debug"}'
          action: drop
      # Sample high-volume logs
      - match:
          selector: '{app="high-volume"}'
          stages:
            - sampling:
                rate: 0.1  # Sample 10% of logs

Common Cause 3: Too Many Unique Streams

Stream count exceeds configured maximum.

Error pattern: `` max streams per user exceeded: creating new stream would exceed limit

Diagnosis:

```bash # Check stream limits curl -s http://localhost:3100/config | jq '.limits.max_streams_per_user'

# Count active streams curl -s http://localhost:3100/metrics | grep loki_ingester_active_streams

# Find label combinations creating streams curl -s 'http://localhost:3100/loki/api/v1/labels' | jq '.data[]'

# Check cardinality per label for label in $(curl -s 'http://localhost:3100/loki/api/v1/labels' | jq -r '.data[]'); do count=$(curl -s "http://localhost:3100/loki/api/v1/label/$label/values" | jq '.data | length') echo "$label: $count unique values" done | sort -t: -k2 -rn | head -20

# Get current stream metrics curl -s http://localhost:3100/metrics | \ grep -E "loki_ingester_streams_created_total|loki_ingester_active_streams" ```

Solution:

Increase stream limit and reduce cardinality:

```yaml # loki-config.yaml limits_config: max_streams_per_user: 100000 # Increase from default max_global_streams_per_user: 500000

# Reduce label cardinality at collection # In Promtail config scrape_configs: - job_name: kubernetes-pods kubernetes_sd_configs: - role: pod pipeline_stages: # Keep only essential labels - labels: app: app_label namespace: namespace # Drop high-cardinality labels - drop_labels: labels: - pod_template_hash - container_id - pod_ip - node

# Use structured metadata instead of labels for high-cardinality data pipeline_stages: - labels: app: nginx - structured_metadata: request_id: request_id # High cardinality as metadata ```

Common Cause 4: Tenant-Specific Rate Limit

Multi-tenant Loki with per-tenant limits exceeded.

Error pattern: `` tenant X ingestion limit exceeded

Diagnosis:

```bash # Check tenant-specific limits curl -s http://localhost:3100/config | jq '.limits.per_tenant_override'

# Find tenant with exceeded limits curl -s http://localhost:3100/metrics | grep -E "loki_distributor.*tenant"

# List tenants if using multi-tenant # Check X-Scope-Orgid header in requests

# Check tenant ingestion metrics curl -s "http://localhost:3100/loki/api/v1/tenant_limits" | jq '.' ```

Solution:

Adjust tenant limits:

```yaml # loki-config.yaml limits_config: ingestion_rate_mb: 20 per_tenant_override_config: /etc/loki/tenant-limits.yaml

# tenant-limits.yaml tenants: tenant-a: ingestion_rate_mb: 50 max_streams_per_user: 50000 tenant-b: ingestion_rate_mb: 30 max_streams_per_user: 30000 ```

Common Cause 5: Ingester Memory Pressure

Ingester memory limits affecting ingestion.

Error pattern: `` ingester memory limit reached, rejecting writes

Diagnosis:

```bash # Check ingester memory metrics curl -s http://localhost:3100/metrics | grep -E "loki_ingester_memory|loki_ingester_chunks"

# Check chunk utilization curl -s http://localhost:3100/metrics | grep loki_ingester_chunks_filled_total

# Check ingester flush status curl -s http://localhost:3100/metrics | grep loki_ingester_flush_queue_length

# Monitor memory usage kubectl top pods -l app=loki -n monitoring

# Check ingester configuration curl -s http://localhost:3100/config | jq '.ingester' ```

Solution:

Optimize ingester memory:

```yaml # loki-config.yaml ingester: lifecycler: ring: kvstore: store: inmemory chunk_idle_period: 5m # Flush sooner chunk_block_size: 256KB chunk_retain_period: 30s max_chunk_age: 2h max_transfer_retries: 0 flush_op_timeout: 10s

limits_config: reject_old_samples: true reject_old_samples_max_age: 168h # 7 days creation_grace_period: 10m

# Increase ingester memory # For Kubernetes deployment resources: limits: memory: 4Gi requests: memory: 2Gi ```

Common Cause 6: Distributor Bottleneck

Distributor cannot handle incoming request volume.

Error pattern: `` distributor rate limit exceeded

Diagnosis:

```bash # Check distributor metrics curl -s http://localhost:3100/metrics | grep -E "loki_distributor"

# Check distributor queue length curl -s http://localhost:3100/metrics | grep loki_distributor_queue_length

# Monitor distributor performance curl -s http://localhost:3100/metrics | \ grep "rate(loki_distributor_requests_total[5m])"

# Check HTTP request errors curl -s http://localhost:3100/metrics | grep loki_distributor_http_requests_failed_total ```

Solution:

Scale distributors or optimize configuration:

```yaml # Scale distributor replicas for load # Kubernetes horizontal pod autoscaler apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: loki-distributor spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: loki-distributor minReplicas: 3 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70

# distributor configuration distributor: ring: kvstore: store: memberlist shards_per_instance: 100 # Increase parallelism ```

Common Cause 7: Promtail Buffer Overflow

Collection side buffer overflow causing rate issues.

Error pattern: `` promtail client buffer full: rate limited

Diagnosis:

```bash # Check Promtail logs kubectl logs -l app=promtail -n monitoring | grep -i "buffer|rate|limit"

# Check Promtail metrics curl -s http://localhost:9080/metrics | grep -E "promtail_lines|promtail_bytes"

# Monitor Promtail buffer curl -s http://localhost:9080/metrics | grep promtail_dropped_bytes_total

# Check batch configuration kubectl describe deployment promtail -n monitoring | grep -A 10 "Args" ```

Solution:

Configure Promtail batching:

```yaml # promtail-config.yaml clients: - url: http://loki:3100/loki/api/v1/push batchwait: 5s # Time before sending batch batchsize: 100KB # Size before sending timeout: 30s max_backoff: 5m max_retries: 10 min_backoff: 500ms

# Increase buffer and retries batch_size: 1048576 # 1MB batch_wait: 10s

# For high-volume logs clients: - url: http://loki:3100/loki/api/v1/push batchsize: 10MB batchwait: 30s max_retries: 20 ```

Verification

After fixing rate limits:

```bash # Check Loki is accepting logs curl -s http://localhost:3100/ready

# Monitor ingestion without errors curl -s http://localhost:3100/metrics | \ grep -E "loki_distributor_lines_received_total|loki_distributor_lines_rejected_total"

# Query recent logs logcli query '{app="nginx"}' --limit=100 --since=10m

# Check no rate limit errors in last 10 minutes kubectl logs -l app=loki -n monitoring --since=10m | grep -c "rate limit"

# Verify configuration applied curl -s http://localhost:3100/config | jq '.limits'

# Test log ingestion manually curl -X POST http://localhost:3100/loki/api/v1/push \ -H "Content-Type: application/json" \ -d '{"streams":[{"stream":{"app":"test"},"values":[["1640000000000000000","test log"]]}]}' ```

Prevention

Set up Loki ingestion monitoring:

```yaml groups: - name: loki_ingestion rules: - alert: LokiIngestionRateLimited expr: rate(loki_distributor_lines_rejected_total[5m]) > 0 for: 5m labels: severity: warning annotations: summary: "Loki rejecting logs due to rate limits"

  • alert: LokiHighIngestionRate
  • expr: rate(loki_distributor_bytes_received_total[5m]) > 50*1024*1024
  • for: 5m
  • labels:
  • severity: warning
  • annotations:
  • summary: "Loki ingestion rate approaching limits"
  • alert: LokiTooManyStreams
  • expr: loki_ingester_active_streams > 100000
  • for: 10m
  • labels:
  • severity: warning
  • annotations:
  • summary: "Loki stream count very high"
  • alert: LokiIngesterMemoryHigh
  • expr: loki_ingester_memory_chunks > 10000
  • for: 5m
  • labels:
  • severity: warning
  • annotations:
  • summary: "Loki ingester memory usage high"
  • `

Regular ingestion health check:

```bash #!/bin/bash # loki-health.sh

# Check rejected logs REJECTED=$(curl -s http://localhost:3100/metrics | \ grep "loki_distributor_lines_rejected_total" | \ awk '{print $2}')

if [ "$REJECTED" -gt 0 ]; then echo "WARNING: $REJECTED logs rejected" # Get rate limit errors kubectl logs -l app=loki --since=5m | grep "rate limit" fi

# Check ingestion rate RATE=$(curl -s http://localhost:3100/metrics | \ grep "rate(loki_distributor_bytes_received_total[5m])" | \ awk '{print $2}') echo "Ingestion rate: $RATE MB/s"

# Check active streams STREAMS=$(curl -s http://localhost:3100/metrics | \ grep "loki_ingester_active_streams" | \ awk '{print $2}') echo "Active streams: $STREAMS" ```

Loki rate limit issues stem from throughput exceeding configured limits, stream cardinality, or per-tenant quotas. Check logs for specific limit exceeded messages, increase limits appropriately, and reduce cardinality through label management.