What's Actually Happening
A resource in your Terraform state is marked as "tainted," meaning Terraform has determined it needs to be destroyed and recreated on the next apply. Tainting can happen automatically when a provisioner fails, or manually when you mark a resource for recreation. Understanding when tainting is appropriate and how to manage it is critical for safe infrastructure updates.
The Error You'll See
When a resource is tainted:
``` # When listing state terraform state list aws_instance.main (tainted)
# When running plan Terraform will perform the following actions:
# aws_instance.main is tainted, so must be replaced -/+ resource "aws_instance" "main" { ~ ami = "ami-0c55b159cbfafe1f0" -> "ami-new" # forces replacement ~ instance_type = "t3.micro" -> "t3.small" # forces replacement id = "i-0123456789abcdef0" } ```
Automatic taint from provisioner failure:
``` Error: Provisioner "remote-exec" failed to execute
The resource aws_instance.main has been marked as tainted and will be destroyed and recreated on the next terraform apply. ```
Why This Happens
Resources become tainted due to:
- 1.Provisioner failure - Terraform automatically taints resources when provisioners fail
- 2.Manual tainting - Using
terraform taintcommand to force recreation - 3.Create failure - Resource partially created but not fully functional
- 4.Health check failure - Provider detects resource is unhealthy
- 5.User intention - Wanting to force recreation of problematic resource
- 6.External modification - Resource modified outside Terraform beyond reconciliation
- 7.Dependency chain - Resource tainted upstream causing downstream taint
Step 1: Identify Tainted Resources
Find which resources are tainted:
```bash # List all resources and see tainted status terraform state list
# Tainted resources show (tainted) after their address: # aws_instance.main (tainted) # aws_subnet.public[0] # module.vpc.aws_vpc.main (tainted)
# Show specific resource details terraform state show aws_instance.main
# The state will show "status: tainted" ```
Plan to see taint effects:
```bash terraform plan
# Tainted resources will show as replacement # -/+ indicates destroy and recreate ```
Step 2: Understand Taint Effects
Tainting affects the entire resource lifecycle:
- 1.
` - 2.When a resource is tainted:
- 3.Terraform plans to DESTROY the existing resource
- 4.Then CREATE a new resource with same configuration
- 5.All dependent resources may also need updates
- 6.Attributes that force replacement will be applied
Example chain: aws_instance.main (tainted) -> destroys i-old, creates i-new aws_ebs_volume.attachment -> must be recreated (depends on instance) ```
Step 3: Manually Taint a Resource
Use tainting to force recreation:
```bash # Basic taint command terraform taint aws_instance.main
# You'll see: # Resource aws_instance.main was marked as tainted.
# Taint specific resource in count/for_each terraform taint 'aws_subnet.public[0]' terraform taint 'aws_subnet.public["us-east-1a"]'
# Taint module resource terraform taint module.vpc.aws_vpc.main terraform taint module.compute.aws_instance.web ```
When to manually taint:
```bash # Resource is corrupted or broken terraform taint aws_instance.broken
# Want to test recreation behavior terraform taint aws_db_instance.test
# Resource needs configuration reset terraform taint aws_security_group.rules
# After debugging, apply to recreate terraform apply ```
Step 4: Remove Taint Mark (Untaint)
When you don't want recreation:
```bash # Remove taint from resource terraform untaint aws_instance.main
# You'll see: # Resource aws_instance.main was successfully untainted.
# Untaint specific index terraform untaint 'aws_subnet.public[0]' terraform untaint 'module.compute.aws_instance.web[0]' ```
When to untaint:
```bash # Resource is actually healthy terraform untaint aws_instance.main
# Provisioner failed but resource is fine terraform untaint aws_instance.main
# Taint was accidental or unnecessary terraform untaint aws_subnet.public[0]
# After fixing underlying issue terraform untaint aws_db_instance.production ```
Step 5: Handle Automatic Taint from Provisioner Failure
When provisioner fails, Terraform automatically taints:
``` # Error during apply Error: provisioner "remote-exec" failed
The resource aws_instance.web has been marked as tainted. ```
Options to handle:
```bash # Option 1: Fix provisioner and untaint # Fix your provisioner script/configuration terraform untaint aws_instance.web terraform apply # Provisioner will run again
# Option 2: Accept recreation # Let Terraform destroy and recreate terraform apply # Will recreate with provisioner
# Option 3: Remove provisioner entirely # Edit configuration to remove provisioner terraform untaint aws_instance.web terraform apply # Resource stays, no provisioner runs ```
Step 6: Handle Taint with Dependencies
Be aware of dependency chains:
```bash # Check dependencies terraform state show aws_instance.main | grep depends_on
# Plan to see cascade effects terraform plan
# If many resources need recreation due to one taint: # Consider untainting if cascade is undesirable terraform untaint aws_instance.main ```
Example of dependency cascade:
```hcl # Tainted instance affects volume resource "aws_instance" "main" { # tainted }
resource "aws_ebs_volume" "data" { instance_id = aws_instance.main.id # Will need recreation too }
# Plan shows: # -/+ aws_instance.main (tainted) # -/+ aws_ebs_volume.data (dependency forces recreation) ```
Step 7: Use Taint for Troubleshooting
Taint helps debug issues:
```bash # Resource behaving strangely, recreate it terraform taint aws_instance.problematic terraform apply
# Test if recreation fixes the issue # If it does, investigate root cause
# After investigation, untaint if recreation wasn't needed terraform untaint aws_instance.problematic ```
Step 8: Handle Taint in CI/CD
Automated handling in pipelines:
```yaml # Example GitHub Actions handling jobs: terraform: steps: - name: Check for tainted resources run: | TAINTED=$(terraform state list | grep tainted) if [ -n "$TAINTED" ]; then echo "::warning::Tainted resources detected: $TAINTED" fi
- name: Terraform Apply
- run: terraform apply -auto-approve
- name: Verify no taints after apply
- run: |
- TAINTED=$(terraform state list | grep tainted)
- if [ -n "$TAINTED" ]; then
- echo "::error::Resources still tainted after apply"
- exit 1
- fi
`
Step 9: Migrate Away from Taint Command
The terraform taint command is deprecated in favor of better approaches:
```hcl # Instead of: terraform taint aws_instance.main # Use: lifecycle replace_triggered_by
resource "aws_instance" "main" { ami = var.ami_id instance_type = var.instance_type
lifecycle { replace_triggered_by = [ var.ami_id, # Replace when AMI changes terraform_data.trigger.version ] } }
resource "terraform_data" "trigger" { lifecycle { replace_triggered_by = [var.force_recreate] } } ```
Alternative approaches:
```hcl # Use replace_triggered_by for controlled recreation lifecycle { replace_triggered_by = [ aws_security_group.main.id ] }
# Use create_before_destroy for safer recreation lifecycle { create_before_destroy = true } ```
Step 10: Handle Specific Taint Scenarios
Instance with data volume:
```bash # If instance tainted, volume data will be lost # Ensure backup before apply aws ec2 create-snapshot --volume-id vol-12345678
terraform taint aws_instance.main terraform apply ```
Database taint (destructive):
```bash # RDS recreation loses all data # NEVER taint production databases casually # Always backup first
aws rds create-db-snapshot \ --db-instance-identifier my-db \ --db-snapshot-identifier my-db-backup
# Then consider taint only if necessary terraform taint aws_db_instance.production ```
Tainted resource blocking plan:
# If taint causes undesirable cascade
terraform plan # Review effects
terraform untaint aws_instance.main # Remove taint
terraform apply # Normal applyVerify the Fix
After handling taint:
```bash # Check no tainted resources remain (if that's goal) terraform state list | grep tainted
# Should return nothing if all untainted
# Or verify tainted resources recreate properly terraform plan terraform apply
# Verify resource recreated successfully terraform state show aws_instance.main ```
Prevention Best Practices
Avoid tainting when possible:
```hcl # Use proper lifecycle management lifecycle { create_before_destroy = true # Safer recreation }
# Use ignore_changes for attributes that fluctuate lifecycle { ignore_changes = [tags["LastModified"]] }
# Use replace_triggered_by instead of manual taint lifecycle { replace_triggered_by = [var.config_version] } ```
Document taint decisions:
## Taint Management Policy
- Never taint production databases without backup
- Review plan before applying taint recreation
- Use untaint for accidental taints
- Prefer lifecycle rules over manual taint