Introduction

Jenkins stores credentials in an encrypted file on the controller filesystem. When a backup process reads or copies these files while Jenkins is actively accessing them, file locks can prevent Jenkins from reading credentials. This causes builds that depend on those credentials -- such as Git checkout, Docker push, or cloud deployment steps -- to fail with authentication errors.

Symptoms

  • Builds fail with Credentials not found or Access denied errors
  • Jenkins logs show java.nio.channels.OverlappingFileLockException
  • Credential management page in Jenkins UI shows errors loading credentials
  • Backup process logs show file access conflicts with Jenkins
  • Error message: com.cloudbees.plugins.credentials.CredentialsUnavailableException: credentials locked

Common Causes

  • Backup tool (rsync, tar, Borg) locking credential files during copy
  • Jenkins and backup process running on the same filesystem with incompatible locking
  • Snapshot-based backup causing credential file state inconsistency
  • Jenkins attempting to read credentials while backup holds an exclusive lock
  • Backup schedule overlapping with peak build activity

Step-by-Step Fix

  1. 1.Identify the locked credential files: Check which files are affected.
  2. 2.```bash
  3. 3.# Find processes holding locks on Jenkins credential files
  4. 4.lsof /var/lib/jenkins/credentials.xml
  5. 5.lsof /var/lib/jenkins/secrets/
  6. 6.`
  7. 7.Stop the conflicting backup process: Release the file lock.
  8. 8.```bash
  9. 9.# Find and stop the backup process
  10. 10.ps aux | grep -i backup
  11. 11.kill <backup-pid>
  12. 12.`
  13. 13.Restart Jenkins to clear the credential store lock: Reload credentials cleanly.
  14. 14.```bash
  15. 15.# Safe restart
  16. 16.curl -X POST https://jenkins.example.com/safeRestart \
  17. 17.-u admin:$(cat /var/lib/jenkins/secrets/initialAdminPassword)
  18. 18.`
  19. 19.Configure backup to use non-locking read mode: Prevent future conflicts.
  20. 20.```bash
  21. 21.# Use rsync with --no-locking option
  22. 22.rsync -av --no-locking /var/lib/jenkins/ /backup/jenkins/

# Or use filesystem snapshots instead of direct file copy # For LVM: lvcreate --snapshot --size 5G --name jenkins-snap /dev/vg0/jenkins mount /dev/vg0/jenkins-snap /mnt/snap rsync -av /mnt/snap/ /backup/jenkins/ umount /mnt/snap lvremove -f /dev/vg0/jenkins-snap ```

  1. 1.Schedule backups during low-activity periods: Avoid overlapping with builds.
  2. 2.```bash
  3. 3.# Run backup during maintenance window (e.g., 2-4 AM)
  4. 4.crontab -e
  5. 5.0 2 * * * /opt/scripts/jenkins-backup.sh
  6. 6.`

Prevention

  • Use filesystem snapshots (LVM, ZFS, EBS) for Jenkins backups instead of direct file copy
  • Schedule backup operations during maintenance windows when no builds are running
  • Configure backup tools to use read-only access without exclusive locks
  • Implement credential caching in Jenkins to reduce file access during builds
  • Monitor credential access errors and alert on credential store lockouts
  • Test backup and restore procedures regularly to ensure credential integrity