A service fails to start. Running systemctl status shows a failed state, but the error message isn't clear enough to identify the problem. systemd is the init system for most modern Linux distributions, and understanding how to troubleshoot it is essential.
Understanding systemd Service States
A service can be in various states:
- active (running) - Service is running normally
- active (exited) - Service ran and completed successfully (one-shot services)
- failed - Service failed to start or crashed
- inactive - Service is stopped
- activating - Service is starting up
Typical Error Messages
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xe" for details.
Failed to start nginx.service: Unit nginx.service not found.
Failed to start application.service: Unit application.service is masked.Diagnosing Service Failures
Basic Diagnostics
```bash # Check service status systemctl status service-name
# Get detailed status with more lines systemctl status service-name -l
# Check if service is enabled systemctl is-enabled service-name
# View service file location systemctl cat service-name ```
Journal Logs
```bash # View logs for a specific service journalctl -u service-name
# View recent logs (last 100 lines) journalctl -u service-name -n 100
# Follow logs in real-time journalctl -u service-name -f
# Logs since specific time journalctl -u service-name --since "1 hour ago" journalctl -u service-name --since "2024-01-15 10:00:00" journalctl -u service-name --since yesterday
# Logs with priority (emerg, alert, crit, err, warning, notice, info, debug) journalctl -u service-name -p err
# Show kernel messages related to service journalctl -k | grep -i service-name
# Export logs for analysis journalctl -u service-name --since "2 hours ago" > service-logs.txt ```
Service Configuration Check
```bash # Verify service file syntax systemd-analyze verify /etc/systemd/system/service-name.service
# Check service dependencies systemctl list-dependencies service-name
# View what a service requires systemctl show service-name -p Requires,Wants,After
# Check for conflicting services systemctl list-units --type=service --all | grep -i conflict ```
Common Issues and Solutions
Issue 1: Service Unit Not Found
```bash # Check if package is installed dpkg -l | grep service-name # Debian/Ubuntu rpm -qa | grep service-name # RHEL/CentOS
# Find service files find /lib/systemd/system -name "*service-name*" find /etc/systemd/system -name "*service-name*"
# Reload systemd to pick up new services systemctl daemon-reload
# If service file is missing, create one cat > /etc/systemd/system/myapp.service << 'EOF' [Unit] Description=My Application After=network.target
[Service] Type=simple User=myapp Group=myapp WorkingDirectory=/opt/myapp ExecStart=/opt/myapp/bin/myapp Restart=on-failure RestartSec=5
[Install] WantedBy=multi-user.target EOF
# Enable and start systemctl daemon-reload systemctl enable myapp systemctl start myapp ```
Issue 2: Service is Masked
A masked service cannot be started:
```bash # Check if masked systemctl is-enabled service-name
# Unmask the service systemctl unmask service-name
# Then start it systemctl start service-name ```
Issue 3: Dependency Failure
```bash # Check dependencies systemctl list-dependencies service-name
# Check failed dependencies systemctl list-dependencies --failed service-name
# Check what services require network systemctl show -p After service-name
# Start required services first systemctl start network.target systemctl start network-online.target
# If dependency is missing, edit service systemctl edit service-name --full ```
Issue 4: Executable Not Found or Not Executable
```bash # Check ExecStart path in service file systemctl cat service-name | grep ExecStart
# Verify executable exists which nginx ls -la /usr/sbin/nginx
# Verify it's executable file /usr/sbin/nginx
# Fix permissions if needed chmod +x /path/to/executable
# Verify shebang for scripts head -1 /path/to/script.sh ```
Issue 5: Permission Denied
```bash # Check service user systemctl cat service-name | grep -E "^(User|Group)"
# Check file permissions ls -la /var/lib/myapp ls -la /var/log/myapp ls -la /var/run/myapp
# Fix ownership chown -R myapp:myapp /var/lib/myapp chown -R myapp:myapp /var/log/myapp
# Check SELinux/AppArmor getenforce # SELinux status aa-status # AppArmor status
# Temporarily disable for testing setenforce 0 # SELinux permissive mode aa-disable /etc/apparmor.d/myapp # Disable AppArmor profile ```
Issue 6: Port Already in Use
```bash # Find what's using the port ss -tlnp | grep :80 netstat -tlnp | grep :80 lsof -i :80
# Check for conflicting services systemctl list-units --type=service | grep -i nginx systemctl list-units --type=service | grep -i apache
# Stop conflicting service systemctl stop apache2 systemctl disable apache2
# Or change port in configuration ```
Issue 7: Configuration File Errors
```bash # Test configuration syntax (application-specific) nginx -t # Nginx apachectl configtest # Apache named-checkconf # BIND sshd -t # SSH
# Check for syntax errors in service file systemd-analyze verify /etc/systemd/system/service-name.service
# View systemd errors journalctl -xe ```
Issue 8: Timeout on Start
```bash # Check current timeout systemctl show service-name -p TimeoutStartUSec
# Extend timeout systemctl edit service-name
[Service] TimeoutStartSec=300
# Or disable timeout TimeoutStartSec=infinity
# Apply changes systemctl daemon-reload systemctl restart service-name ```
Issue 9: Service Crashes After Start
```bash # Check exit code systemctl status service-name
# Common exit codes: # 1 - General error # 2 - Misuse of shell command # 126 - Cannot execute # 127 - Command not found # 128+N - Signal N (128+9=137 = SIGKILL) # 130 - Ctrl+C (128+2)
# Check for OOM killer dmesg | grep -i oom journalctl -k | grep -i killed
# Increase resource limits systemctl edit service-name
[Service] LimitNOFILE=65535 LimitNPROC=4096 LimitMEMLOCK=infinity ```
Issue 10: Environment Variable Issues
```bash # Check environment variables in service systemctl show service-name -p Environment
# Add environment variables systemctl edit service-name
[Service] Environment="NODE_ENV=production" Environment="DATABASE_URL=postgres://localhost/db" EnvironmentFile=/etc/myapp/config.env
# Or use drop-in directory mkdir -p /etc/systemd/system/service-name.service.d cat > /etc/systemd/system/service-name.service.d/override.conf << 'EOF' [Service] Environment="VAR1=value1" Environment="VAR2=value2" EOF
systemctl daemon-reload systemctl restart service-name ```
Debug Mode
Enable debug logging for systemd:
```bash # Enable systemd debug logging mkdir -p /etc/systemd/system.conf.d cat > /etc/systemd/system.conf.d/debug.conf << 'EOF' [Manager] LogLevel=debug LogTarget=journal EOF
systemctl daemon-reexec
# Debug specific service start SYSTEMD_LOG_LEVEL=debug systemctl start service-name ```
Run service interactively for debugging:
```bash # Get the exact command the service runs systemctl cat service-name | grep ExecStart
# Run as the service user sudo -u service-user /path/to/executable arguments
# Check environment sudo -u service-user env
# Check working directory sudo -u service-user pwd ```
Service File Best Practices
A well-structured service file:
```ini [Unit] Description=My Application Service Documentation=https://example.com/docs After=network-online.target Wants=network-online.target ConditionPathExists=/etc/myapp/config.yml
[Service] Type=simple User=myapp Group=myapp WorkingDirectory=/opt/myapp ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yml ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure RestartSec=5 TimeoutStartSec=30 TimeoutStopSec=30
# Security hardening NoNewPrivileges=true PrivateTmp=true ProtectSystem=strict ProtectHome=true ReadWritePaths=/var/lib/myapp /var/log/myapp
# Resource limits LimitNOFILE=65535 LimitNPROC=4096
# Environment Environment="NODE_ENV=production" EnvironmentFile=/etc/myapp/env
[Install] WantedBy=multi-user.target ```
Verification
After fixing issues, verify the service works correctly:
```bash # Start service systemctl start service-name
# Check status systemctl status service-name
# Verify it's running systemctl is-active service-name
# Check if enabled at boot systemctl is-enabled service-name
# View recent logs journalctl -u service-name -n 50
# Test service functionality curl http://localhost:port netstat -tlnp | grep service-name
# Verify process is running ps aux | grep service-name pgrep -a service-name ```