Introduction

When a Helm chart uses the latest tag for container images in its default values, the actual image deployed changes whenever the upstream registry updates the latest tag. This causes deployment drift -- two identical helm upgrade commands with the same values can produce different running containers. This makes debugging, rollback, and reproducibility nearly impossible.

Symptoms

  • Pod behavior changes between deployments without any Helm values changes
  • Rollback to a previous Helm revision does not restore the previous application version
  • kubectl describe pod shows different image digests for the same image tag
  • Different cluster environments running different application versions with the same Helm chart
  • Error message: ImagePullBackOff when latest tag is removed from the registry

Common Causes

  • Default values.yaml sets image.tag: latest
  • CI/CD pipeline does not update the image tag during deployment
  • Chart author not aware of the risks of mutable image tags
  • Development workflow using latest for convenience, leaked to production
  • Image registry auto-promoting latest on every push to the main branch

Step-by-Step Fix

  1. 1.Identify charts using the latest tag: Find all occurrences of latest.
  2. 2.```bash
  3. 3.grep -r "tag:.*latest|tag: \"latest\"" ./charts/*/values.yaml
  4. 4.`
  5. 5.Pin the image tag to a specific version: Use semantic versioning.
  6. 6.```yaml
  7. 7.# values.yaml
  8. 8.image:
  9. 9.repository: myapp
  10. 10.tag: "2.3.1" # NOT "latest"
  11. 11.pullPolicy: IfNotPresent
  12. 12.`
  13. 13.Use image digest for maximum reproducibility: Pin to a specific image hash.
  14. 14.```yaml
  15. 15.image:
  16. 16.repository: myapp
  17. 17.tag: "2.3.1"
  18. 18.digest: "sha256:abc123def456..."
  19. 19.pullPolicy: IfNotPresent
  20. 20.`
  21. 21.Update CI/CD pipeline to set the image tag at deploy time: Pass the tag from the build.
  22. 22.```bash
  23. 23.# In CI/CD pipeline
  24. 24.IMAGE_TAG=$(cat image-tag.txt)
  25. 25.helm upgrade my-release ./chart \
  26. 26.--set image.tag="$IMAGE_TAG" \
  27. 27.--namespace production
  28. 28.`
  29. 29.Add image tag validation to the chart: Prevent latest from being used.
  30. 30.```yaml
  31. 31.# templates/_helpers.tpl
  32. 32.{{- if eq .Values.image.tag "latest" }}
  33. 33.{{- fail "Image tag 'latest' is not allowed. Please specify a specific version." }}
  34. 34.{{- end }}
  35. 35.`

Prevention

  • Never use latest as a default image tag in Helm charts -- always pin to a specific version
  • Add helm lint rules that fail on latest image tags
  • Use image digests in addition to tags for production deployments
  • Configure CI/CD pipelines to automatically update image tags during deployment
  • Use a container image registry that supports image promotion workflows instead of mutable tags
  • Audit running pods periodically to detect image tag drift across environments