The Problem

The Thanos sidecar running alongside Prometheus is failing to upload blocks or provide query access. You see errors like:

bash
level=error ts=2026-04-04T22:40:15.234Z caller=sidecar.go:123 msg="upload block failed" err="context canceled"
level=error ts=2026-04-04T22:40:16.345Z caller=sidecar.go:124 msg="failed to upload block" block="01HXYZ..." err="access denied"
level=warn ts=2026-04-04T22:40:17.456Z caller=sidecar.go:125 msg="Prometheus unreachable" err="dial tcp 127.0.0.1:9090: connection refused"
level=error ts=2026-04-04T22:40:18.567Z caller=objstore.go:234 msg="error accessing bucket" err="NoSuchBucket: The specified bucket does not exist"

Thanos sidecar errors prevent long-term metric storage and query access to historical data.

Diagnosis

Check Thanos Sidecar Logs

```bash # Check Thanos sidecar logs kubectl logs prometheus-0 -c thanos-sidecar --tail=100

# Or for standalone journalctl -u thanos-sidecar --since "1 hour ago" ```

Check Thanos Sidecar Metrics

```promql # Thanos sidecar upload status thanos_sidecar_upload_requests_total

# Failed uploads rate(thanos_sidecar_upload_requests_failures_total[5m])

# Blocks uploaded thanos_sidecar_blocks_uploaded_total

# Object store operations thanos_objstore_bucket_operations_total ```

Check Object Storage Connectivity

```bash # Test S3 bucket access (example) aws s3 ls s3://thanos-bucket/

# Test GCS bucket access gsutil ls gs://thanos-bucket/

# Test MinIO mc ls minio/thanos-bucket/

# Use thanos tools to test thanos tools bucket ls --objstore.config-file=bucket.yaml ```

Check Prometheus Connection

```bash # Verify Prometheus is accessible curl -s http://prometheus:9090/-/healthy

# Check Prometheus external labels curl -s http://prometheus:9090/api/v1/status/config | jq '.data.global.external_labels' ```

Solutions

1. Fix Object Storage Configuration

Incorrect bucket configuration:

yaml
# bucket.yaml - Object storage config
type: S3
config:
  bucket: thanos-bucket
  endpoint: s3.amazonaws.com
  region: us-east-1
  access_key: AKIAIOSFODNN7EXAMPLE
  secret_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  # For MinIO or custom S3
  # endpoint: minio.example.com:9000
  # insecure: true

For Thanos sidecar startup:

bash
thanos sidecar \
  --objstore.config-file=bucket.yaml \
  --prometheus.url=http://prometheus:9090

Fix bucket permissions:

```bash # Create bucket if missing aws s3 mb s3://thanos-bucket

# Set lifecycle policy aws s3api put-bucket-lifecycle-configuration \ --bucket thanos-bucket \ --lifecycle-configuration file://lifecycle.json ```

2. Fix Authentication Errors

Access denied to object storage:

yaml
# bucket.yaml with correct credentials
type: S3
config:
  bucket: thanos-bucket
  endpoint: s3.amazonaws.com
  access_key: ${AWS_ACCESS_KEY_ID}
  secret_key: ${AWS_SECRET_ACCESS_KEY}
  session_token: ${AWS_SESSION_TOKEN}  # If using temporary credentials

For Kubernetes with IAM roles:

yaml
# bucket.yaml
type: S3
config:
  bucket: thanos-bucket
  endpoint: s3.amazonaws.com
  # No explicit credentials - use IAM role
yaml
# pod annotation for IAM role
metadata:
  annotations:
    iam.amazonaws.com/role: arn:aws:iam::123456789:role/thanos-role

3. Fix Prometheus Connection

Sidecar cannot reach Prometheus:

bash
thanos sidecar \
  --prometheus.url=http://localhost:9090 \
  --prometheus.http-client-config-file=http-client.yaml
yaml
# http-client.yaml
tls_config:
  ca_file: /etc/thanos/certs/ca.crt
  cert_file: /etc/thanos/certs/client.crt
  key_file: /etc/thanos/certs/client.key
  # insecure_skip_verify: true
basic_auth:
  username: thanos
  password: thanos_password

Kubernetes sidecar configuration:

yaml
# sidecar container alongside Prometheus
containers:
  - name: prometheus
    image: prom/prometheus:latest
    ports:
      - containerPort: 9090
  - name: thanos-sidecar
    image: thanosio/thanos:latest
    args:
      - sidecar
      - --prometheus.url=http://localhost:9090
      - --objstore.config-file=/etc/thanos/bucket.yaml
      - --grpc-address=0.0.0.0:10901
      - --http-address=0.0.0.0:10902
    ports:
      - containerPort: 10901
      - containerPort: 10902
    volumeMounts:
      - name: thanos-config
        mountPath: /etc/thanos

4. Fix External Labels

Thanos requires external labels for deduplication:

yaml
# Prometheus configuration
global:
  external_labels:
    cluster: 'production'
    prometheus: 'prometheus-0'
    replica: '0'

If missing, add via Thanos sidecar flags:

bash
thanos sidecar \
  --prometheus.url=http://localhost:9090 \
  --objstore.config-file=bucket.yaml \
  --shipper.label='cluster="production"' \
  --shipper.label='prometheus="prometheus-0"'

5. Fix Upload Timing Issues

Blocks not being uploaded timely:

bash
thanos sidecar \
  --prometheus.url=http://localhost:9090 \
  --objstore.config-file=bucket.yaml \
  --shipper.upload-interval=5m  # How often to check for new blocks

Check block compaction:

```promql # Prometheus block status prometheus_tsdb_compactions_total prometheus_tsdb_head_series

# Thanos shipper status thanos_shipper_upload_success_total ```

6. Fix Query Gateway Integration

Thanos Query cannot reach sidecar:

bash
# Thanos Query configuration
thanos query \
  --grpc-address=0.0.0.0:10901 \
  --http-address=0.0.0.0:10902 \
  --store=thanos-sidecar-0:10901 \
  --store=thanos-sidecar-1:10901 \
  --query.replica-label=prometheus

Kubernetes service for sidecar:

yaml
apiVersion: v1
kind: Service
metadata:
  name: thanos-sidecar-grpc
spec:
  type: ClusterIP
  ports:
    - port: 10901
      name: grpc
  selector:
    app: prometheus

7. Handle WAL Errors

Issues with WAL shipping:

bash
thanos sidecar \
  --prometheus.url=http://localhost:9090 \
  --objstore.config-file=bucket.yaml \
  # WAL shipping is optional
  --shipper.upload-interval=5m

If WAL shipping is enabled:

yaml
# Enable WAL shipping for near-real-time
thanos sidecar \
  --prometheus.url=http://localhost:9090 \
  --objstore.config-file=bucket.yaml \
  --shipper.ship-wal=true \
  --shipper.upload-interval=30s

Verification

Check Block Uploads

```bash # List blocks in bucket thanos tools bucket ls --objstore.config-file=bucket.yaml

# View block metadata thanos tools bucket view --objstore.config-file=bucket.yaml ```

Test Query Access

```bash # Query via Thanos Query curl -s 'http://thanos-query:10902/api/v1/query?query=up' | jq .

# Check connected stores curl -s 'http://thanos-query:10902/api/v1/stores' | jq . ```

Verify Metrics

```promql # Thanos sidecar health thanos_sidecar_ready == 1

# Successful uploads rate(thanos_sidecar_upload_requests_success_total[5m]) > 0

# Object store operations rate(thanos_objstore_bucket_operations_total{operation="upload"}[5m]) ```

Prevention

Add monitoring for Thanos sidecar:

```yaml groups: - name: thanos_sidecar_alerts rules: - alert: ThanosSidecarUnhealthy expr: thanos_sidecar_ready != 1 for: 5m labels: severity: critical annotations: summary: "Thanos sidecar is unhealthy" description: "Thanos sidecar {{ $labels.instance }} is not ready"

  • alert: ThanosSidecarUploadFailing
  • expr: rate(thanos_sidecar_upload_requests_failures_total[5m]) > 0
  • for: 5m
  • labels:
  • severity: critical
  • annotations:
  • summary: "Thanos sidecar upload failing"
  • description: "{{ $value }} uploads failed per second"
  • alert: ThanosSidecarNoUploads
  • expr: rate(thanos_sidecar_upload_requests_total[30m]) == 0
  • for: 30m
  • labels:
  • severity: warning
  • annotations:
  • summary: "Thanos sidecar not uploading blocks"
  • description: "No block uploads in last 30 minutes"
  • alert: ThanosBucketAccessError
  • expr: rate(thanos_objstore_bucket_operations_failures_total[5m]) > 0
  • for: 5m
  • labels:
  • severity: critical
  • annotations:
  • summary: "Object storage access errors"
  • description: "{{ $value }} bucket operations failed per second"
  • alert: ThanosSidecarPrometheusUnreachable
  • expr: thanos_sidecar_prometheus_up != 1
  • for: 5m
  • labels:
  • severity: critical
  • annotations:
  • summary: "Thanos sidecar cannot reach Prometheus"
  • `