Introduction

Helm v3 stores release information as Kubernetes Secrets in the deployment namespace. Each secret contains the full rendered chart manifest. When a chart produces large manifests -- due to many resources, large ConfigMaps, or embedded data -- the combined release secret can exceed etcd's maximum object size (typically 1MB), causing the upgrade to fail.

Symptoms

  • helm upgrade fails with Secret is invalid or etcd request too large
  • Kubernetes API returns Request entity too large for the release secret
  • Charts with many resources or large ConfigMaps fail to deploy
  • Error message: Operation cannot be fulfilled on secrets: the object has been modified
  • Error message: etcdserver: request is too large

Common Causes

  • Chart includes large ConfigMaps with embedded configuration data
  • Many Kubernetes resources in a single chart (hundreds of deployments, services, etc.)
  • Release history accumulating multiple large release secrets
  • Certificates or binary data embedded in ConfigMaps
  • Helm not compressing release data in older versions

Step-by-Step Fix

  1. 1.Check the release secret size: Identify how large the secret has grown.
  2. 2.```bash
  3. 3.kubectl get secret -n my-namespace -l owner=helm -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.data}{"\n"}{end}' | awk '{print $1, length($2)}'
  4. 4.`
  5. 5.Reduce the number of resources in the chart: Split large charts into smaller sub-charts.
  6. 6.```bash
  7. 7.# Split monolithic chart into smaller focused charts
  8. 8.helm install my-app-core ./charts/core -n my-namespace
  9. 9.helm install my-app-addons ./charts/addons -n my-namespace
  10. 10.`
  11. 11.Move large ConfigMap data to external storage: Reference data from ConfigMaps stored separately.
  12. 12.```yaml
  13. 13.# Instead of embedding large data in the chart
  14. 14.# Use external ConfigMap
  15. 15.configMapRef:
  16. 16.name: large-config # Created outside the Helm chart
  17. 17.`
  18. 18.Enable Helm release data compression: Ensure release data is compressed.
  19. 19.```bash
  20. 20.# Helm 3.5+ compresses release data by default
  21. 21.# For older versions, upgrade Helm client
  22. 22.helm version
  23. 23.`
  24. 24.Clean up old release history: Remove old release secrets to free space.
  25. 25.```bash
  26. 26.helm history my-release -n my-namespace
  27. 27.# Set max history in the chart or via environment
  28. 28.export HELM_MAX_HISTORY=5
  29. 29.`

Prevention

  • Keep individual chart resource count below 100 to avoid oversized release secrets
  • Store large configuration data in external ConfigMaps or secrets managed separately from Helm
  • Use helm install --atomic to prevent partial releases that create orphaned resources
  • Set HELM_MAX_HISTORY to limit the number of release secrets retained
  • Split monolithic applications into multiple Helm releases using dependency management
  • Monitor release secret sizes and alert when they approach the 1MB etcd limit