Introduction

When GitHub Actions secrets (AWS credentials, deployment tokens, API keys) are rotated, the new values may differ in format, permissions, or associated configuration from the old ones. Deployment scripts that depend on specific secret formats or associated infrastructure configuration can fail immediately after rotation, blocking production deployments.

Symptoms

  • Deployment workflow fails immediately after secret rotation
  • Error messages indicate authentication or authorization failures with new credentials
  • Old secret format was JSON but new secret is a plain string
  • Deployment script expects a specific key prefix or structure in the secret
  • Error message: The security token included in the request is invalid

Common Causes

  • New credentials generated with different IAM permissions than the old ones
  • Secret format changed (e.g., access key ID format, token structure)
  • Deployment script hardcoded assumptions about the secret value format
  • Rotated secret requires associated infrastructure changes (e.g., new SSH key added to server)
  • Multiple secrets that need to be rotated together but only some were updated

Step-by-Step Fix

  1. 1.Identify which rotated secret is causing the failure: Check the workflow logs.
  2. 2.```yaml
  3. 3.# In the workflow run logs, look for the failing step
  4. 4.# Check which secret is being used: ${{ secrets.AWS_ACCESS_KEY_ID }}
  5. 5.# Note the specific error message
  6. 6.`
  7. 7.Verify the new secret has the correct permissions: Test the credentials.
  8. 8.```yaml
  9. 9.- name: Test new credentials
  10. 10.run: |
  11. 11.aws sts get-caller-identity
  12. 12.env:
  13. 13.AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  14. 14.AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  15. 15.`
  16. 16.Update the deployment script to handle the new secret format: Fix format assumptions.
  17. 17.```bash
  18. 18.# BEFORE: Expected base64 encoded JSON
  19. 19.DEPLOY_TOKEN=$(echo $DEPLOY_SECRET | base64 -d | jq -r '.token')

# AFTER: Handle both formats if echo "$DEPLOY_SECRET" | base64 -d 2>/dev/null | jq . >/dev/null 2>&1; then DEPLOY_TOKEN=$(echo "$DEPLOY_SECRET" | base64 -d | jq -r '.token') else DEPLOY_TOKEN="$DEPLOY_SECRET" fi ```

  1. 1.Update associated infrastructure if needed: Ensure the new credentials work end-to-end.
  2. 2.```bash
  3. 3.# If rotating SSH keys, update the authorized_keys on target servers
  4. 4.ansible all -m authorized_key -a "user=deploy key='{{ new_ssh_public_key }}'"
  5. 5.`
  6. 6.Test the deployment with the new secrets in staging: Verify before production.
  7. 7.```bash
  8. 8.# Trigger the workflow on a staging branch
  9. 9.gh workflow run deploy.yml --ref staging
  10. 10.`

Prevention

  • Test secret rotation in a staging environment before applying to production
  • Document the expected format and associated configuration for each secret
  • Use credential formats that are self-describing (e.g., JSON with metadata)
  • Implement secret rotation as an automated process that updates all dependent configuration
  • Add pre-deployment credential validation steps to the workflow
  • Use short-lived credentials (OIDC, temporary tokens) to reduce the impact of rotation