Introduction

Jenkins loads each plugin in its own classloader. When plugins are updated or reloaded, the old classloaders should be garbage collected. However, classloader leaks occur when references to the old classloaders are retained by static fields, threads, or external libraries. Over time, this exhausts the JVM's Metaspace (or PermGen in older JVMs), causing the Jenkins master to crash with an OutOfMemoryError.

Symptoms

  • Jenkins master crashes with java.lang.OutOfMemoryError: Metaspace or PermGen space
  • Jenkins becomes increasingly slow over days or weeks before crashing
  • jstat -gcmetacapacity shows Metaspace usage approaching the maximum
  • Plugin hot-reloads correlate with Metaspace growth
  • Error message: java.lang.OutOfMemoryError: Metaspace in Jenkins logs

Common Causes

  • Plugin classloader not being garbage collected after plugin update
  • Static references in plugins holding references to classloaders
  • Thread-local variables in plugins not cleaned up during reload
  • Too many plugins installed, each with its own classloader
  • JVM Metaspace size set too low for the number of installed plugins

Step-by-Step Fix

  1. 1.Check JVM memory usage and Metaspace configuration: Verify the current Metaspace limits.
  2. 2.```bash
  3. 3.# Check Jenkins JVM arguments
  4. 4.grep -r "MaxMetaspaceSize|MaxPermSize" /etc/default/jenkins /etc/sysconfig/jenkins
  5. 5.# Check current Metaspace usage
  6. 6.jstat -gcmetacapacity $(pgrep -f jenkins.war)
  7. 7.`
  8. 8.Increase Metaspace size: Allocate more memory for class metadata.
  9. 9.```bash
  10. 10.# /etc/default/jenkins or /etc/sysconfig/jenkins
  11. 11.JAVA_ARGS="$JAVA_ARGS -XX:MaxMetaspaceSize=512m"
  12. 12.JAVA_ARGS="$JAVA_ARGS -XX:MetaspaceSize=256m"
  13. 13.# Restart Jenkins
  14. 14.systemctl restart jenkins
  15. 15.`
  16. 16.Identify and remove unnecessary plugins: Reduce classloader count.
  17. 17.`
  18. 18.# Jenkins UI: Manage Jenkins > Manage Plugins > Installed
  19. 19.# Identify plugins that are not used by any job
  20. 20.# Uninstall unused plugins
  21. 21.`
  22. 22.Enable classloader leak detection: Monitor for leaking classloaders.
  23. 23.```bash
  24. 24.# Add JVM flags for Metaspace monitoring
  25. 25.JAVA_ARGS="$JAVA_ARGS -XX:+TraceClassUnloading"
  26. 26.JAVA_ARGS="$JAVA_ARGS -verbose:class"
  27. 27.# Restart and monitor logs
  28. 28.`
  29. 29.Schedule regular Jenkins restarts: Prevent Metaspace accumulation.
  30. 30.```bash
  31. 31.# Weekly restart during maintenance window
  32. 32.crontab -e
  33. 33.0 3 * * 0 systemctl restart jenkins
  34. 34.# Or use Jenkins Safe Restart
  35. 35.curl -X POST https://jenkins.example.com/safeRestart -u admin:token
  36. 36.`

Prevention

  • Set -XX:MaxMetaspaceSize=512m or higher for Jenkins with many plugins
  • Minimize the number of installed plugins to reduce classloader overhead
  • Monitor Metaspace usage with JMX or Prometheus JMX exporter
  • Schedule regular Jenkins restarts during maintenance windows
  • Use Jenkins LTS (Long Term Support) releases which include classloader leak fixes
  • Replace hot-reload plugin updates with full Jenkins restarts to ensure clean classloader unload