Introduction

A failed deployment can leave an application in a partially updated state where some files are from the new version and others are from the old version. This inconsistency causes import errors, missing functions, database schema mismatches, and mixed API responses. The site may return 500 errors, serve broken pages, or crash entirely. Without a proper rollback strategy, recovering from a failed deployment can be time-consuming and risky.

Symptoms

  • Application returns 500 Internal Server Error immediately after deployment
  • Error logs show ModuleNotFoundError, undefined method, or table does not exist
  • Some pages work while others fail (inconsistent deployment state)
  • Database migrations partially applied (some tables updated, others not)
  • Frontend assets (CSS/JS) mismatched with backend API responses

Common Causes

  • Deployment script interrupted mid-execution (network timeout, SSH disconnect)
  • Database migration failed halfway through a multi-step migration
  • Asset compilation (webpack, esbuild) failed after code was deployed
  • Blue-green or canary deployment not properly configured
  • File transfer (rsync, scp) incomplete due to disk space or network issues

Step-by-Step Fix

  1. 1.Identify the deployment state:
  2. 2.```bash
  3. 3.# Check the current deployed version
  4. 4.cat /var/www/myapp/VERSION 2>/dev/null
  5. 5.# Check git status if deploying from git
  6. 6.cd /var/www/myapp && git log -1 --oneline
  7. 7.# Check if migration is in progress
  8. 8.cat /var/www/myapp/tmp/migration.lock 2>/dev/null
  9. 9.`
  10. 10.Rollback to the previous known-good deployment:
  11. 11.```bash
  12. 12.# If using Capistrano:
  13. 13.cd /var/www/myapp && cap production deploy:rollback

# If using a symlink-based deployment (common pattern): ls -la /var/www/myapp/current # Shows which release is active ls -d /var/www/myapp/releases/* # List all releases ln -sfn /var/www/myapp/releases/previous_release /var/www/myapp/current sudo systemctl restart myapp ```

  1. 1.Rollback database migrations if needed:
  2. 2.```bash
  3. 3.# Rails
  4. 4.cd /var/www/myapp && RAILS_ENV=production bundle exec rake db:rollback

# Django cd /var/www/myapp && python manage.py migrate app_name previous_migration

# Laravel cd /var/www/myapp && php artisan migrate:rollback ```

  1. 1.If no rollback is available, restore from backup:
  2. 2.```bash
  3. 3.# Restore code from backup
  4. 4.sudo tar -xzf /backup/myapp-$(date -d "yesterday" +%Y%m%d).tar.gz -C /var/www/
  5. 5.# Restore database from backup
  6. 6.sudo -u postgres pg_restore -d mydb /backup/mydb-$(date -d "yesterday" +%Y%m%d).sql
  7. 7.sudo systemctl restart myapp
  8. 8.`
  9. 9.Fix the deployment and retry:
  10. 10.```bash
  11. 11.# Diagnose why the deployment failed
  12. 12.# Check deployment logs
  13. 13.cat /var/log/deploy.log
  14. 14.# Fix the issue and redeploy
  15. 15.`
  16. 16.Implement atomic deployments to prevent partial updates:
  17. 17.```bash
  18. 18.# Use the symlink pattern:
  19. 19.# 1. Deploy to a new release directory
  20. 20.# 2. Run migrations
  21. 21.# 3. Compile assets
  22. 22.# 4. Atomically switch the symlink
  23. 23.# 5. Restart the application
  24. 24.# If any step fails, the old release is still active
  25. 25.`

Prevention

  • Use atomic deployment strategies (symlink switching, blue-green, containers)
  • Implement health checks that verify the deployment before switching traffic
  • Always maintain at least one previous release for instant rollback
  • Run database migrations as a separate, reversible step
  • Test deployments in a staging environment that mirrors production
  • Use CI/CD pipelines with automatic rollback on health check failure