Introduction

CloudWatch Logs subscription filters can look healthy in the console while silently forwarding nothing. In practice, the pipeline usually breaks in one of three places: the filter matches no events, the destination refuses invocations, or the destination is throttled hard enough that CloudWatch Logs drops or delays delivery.

Symptoms

  • New log events appear in the log group, but the Lambda, Kinesis, or Firehose destination receives nothing
  • DeliveryThrottling or delivery error metrics rise while IncomingLogEvents continues increasing
  • The subscription filter exists and shows the expected destination ARN
  • A downstream SIEM, alerting pipeline, or analytics stream suddenly stops updating

Common Causes

  • The filter pattern is too restrictive and matches no incoming events
  • Lambda or Kinesis destination permissions do not trust the correct CloudWatch Logs service principal
  • Cross-account delivery policy or destination ARN configuration drifted
  • The destination is throttled or undersized for the current log volume

Step-by-Step Fix

  1. 1.Inspect the active subscription filter and destination
  2. 2.Confirm there is only the expected filter and that it targets the correct region, account, and destination ARN.
bash
aws logs describe-subscription-filters \
  --log-group-name /aws/lambda/my-function \
  --query "subscriptionFilters[*].[filterName,filterPattern,destinationArn]"
  1. 1.Temporarily test with an empty filter pattern
  2. 2.An empty pattern matches all events. If delivery starts working after this change, the transport path is healthy and the original pattern was the problem.
bash
aws logs put-subscription-filter \
  --log-group-name /aws/lambda/my-function \
  --filter-name ship-all-events \
  --filter-pattern "" \
  --destination-arn arn:aws:lambda:us-east-1:123456789012:function:log-processor
  1. 1.Verify destination permissions for the exact CloudWatch Logs principal
  2. 2.Lambda destinations need a resource policy that trusts logs.<region>.amazonaws.com. Cross-account targets need matching destination policy on both sides.

```bash aws lambda get-policy --function-name log-processor

aws lambda add-permission \ --function-name log-processor \ --statement-id cwlogs-us-east-1 \ --action lambda:InvokeFunction \ --principal logs.us-east-1.amazonaws.com \ --source-arn "arn:aws:logs:us-east-1:123456789012:log-group:/aws/lambda/my-function:*" ```

  1. 1.Check delivery throttling on the destination
  2. 2.If the destination is Kinesis or Firehose, inadequate shard or throughput capacity can stall forwarding even when the filter and policy are correct.
bash
aws cloudwatch get-metric-statistics \
  --namespace AWS/Logs \
  --metric-name DeliveryThrottling \
  --start-time 2026-04-10T03:30:00Z \
  --end-time 2026-04-10T04:30:00Z \
  --period 300 \
  --statistics Sum

Prevention

  • Keep subscription filters simple and test them with representative sample events
  • Manage Lambda permissions and cross-account destination policies with infrastructure as code
  • Monitor delivery throttling and downstream backlog metrics together
  • Re-validate log forwarding after region changes, account moves, or destination replacement