Introduction StatefulSets with persistent storage are the most challenging workloads to migrate between Kubernetes clusters. If persistent volumes are not properly migrated, stateful applications lose data.

Symptoms - StatefulSet pods starting with empty data directories - Database StatefulSet showing no existing data - PVCs bound to non-existent PVs after migration - Application data missing after cluster migration - StatefulSet ordering violated during migration

Common Causes - PVs not migrated (cloud provider specific storage) - PVC not properly rebound to new PV - StatefulSet migrated before data transfer complete - StorageClass not available in new cluster - Data transfer incomplete before pod startup

Step-by-Step Fix 1. **Backup stateful data before migration': ```bash # Using Velero velero backup create pre-migration --include-namespaces production # Verify backup velero backup describe pre-migration --details ```

  1. 1.**Migrate data first, then StatefulSet':
  2. 2.```bash
  3. 3.# Restore data to new cluster storage
  4. 4.velero restore create --from-backup pre-migration
  5. 5.# Then deploy StatefulSet
  6. 6.kubectl apply -f statefulset.yaml
  7. 7.`
  8. 8.**Verify data integrity':
  9. 9.```bash
  10. 10.kubectl exec <statefulset-pod> -- ls /data/
  11. 11.kubectl exec <statefulset-pod> -- wc -l /data/*
  12. 12.`

Prevention - Use portable storage (NFS, Ceph) for StatefulSets - Backup all stateful data before cluster migration - Migrate data before deploying StatefulSets - Test migration with non-production StatefulSets first - Use Velero for cluster-to-cluster backup and restore