Fix Systemd Service Failed to Start Automatically

What's Actually Happening

You configured a systemd service to start automatically on boot or restart when it fails, but it's not working as expected. The service shows as "failed" status, doesn't start when the system boots, crashes immediately after starting, or won't restart when you expect it to.

Systemd is the init system and service manager for most modern Linux distributions. When services don't start automatically, it affects application availability, scheduled tasks, and system reliability. This commonly happens after system updates, configuration changes, or when deploying new services.

The Error You'll See

Systemd service failures manifest in various ways:

```bash # Service failed status $ systemctl status myapp ● myapp.service - My Application Service Loaded: loaded (/etc/systemd/system/myapp.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2026-04-08 10:15:23 UTC; 5s ago Process: 12345 ExecStart=/usr/bin/myapp (code=exited, status=1/FAILURE) Main PID: 12345 (code=exited, status=1/FAILURE)

Apr 08 10:15:23 server myapp[12345]: Error: Cannot bind to address 0.0.0.0:8080 Apr 08 10:15:23 server systemd[1]: myapp.service: Failed with result 'exit-code'. Apr 08 10:15:23 server systemd[1]: Failed to start My Application Service.

# Service not starting on boot $ systemctl is-enabled myapp disabled

$ systemctl list-unit-files | grep myapp myapp.service disabled

# Service starting but crashing immediately $ systemctl status myapp ● myapp.service - My Application Service Active: activating (auto-restart) since Mon 2026-04-08 10:15:30 UTC; 2s ago Process: 12346 ExecStart=/usr/bin/myapp (code=exited, status=127/COMMAND NOT FOUND)

# Service stuck in activating state $ systemctl status myapp ● myapp.service - My Application Service Active: activating (start) since Mon 2026-04-08 10:15:00 UTC; 5min ago

# Service not restarting on failure # Restart=on-failure in unit file but service stays stopped after crash

# Dependency ordering issues $ systemctl start myapp Job for myapp.service failed because a dependency's unit file was not found. Dependency: network-online.target ```

Additional symptoms: - Service shows "inactive (dead)" after boot - systemctl enable doesn't create symlinks - Service starts manually but not automatically - Service crashes and doesn't restart despite Restart=on-failure - Multiple service instances running unexpectedly - Error messages about missing ExecStart command - Permission denied errors when service tries to run

Why This Happens

1. 1.Service Not Enabled: The service unit file exists but wasn't enabled with systemctl enable. Without being enabled, systemd doesn't create the symlinks needed to start the service during boot.
2. 2.Incorrect Unit File Configuration: The service unit file has syntax errors, wrong paths, missing required sections, or incorrect directives. Common issues: wrong ExecStart path, missing User/Group, wrong permissions on the executable.
3. 3.Missing Dependencies: The service depends on other services or targets that aren't available, haven't started yet, or don't exist. After= and Requires= directives reference non-existent units.
4. 4.Permission Issues: The service tries to access files, directories, or ports that the configured user doesn't have permission for. Executable not marked as executable, wrong file ownership, or SELinux/AppArmor blocking access.
5. 5.ExecStart Command Fails: The command specified in ExecStart doesn't exist, has wrong arguments, or exits immediately with an error. The application fails to start due to configuration errors, missing dependencies, or resource issues.
6. 6.Working Directory Issues: The service tries to run from a directory that doesn't exist or isn't accessible. Relative paths in ExecStart fail because the working directory is wrong.
7. 7.Type= Mismatch: The service Type is set incorrectly. For example, Type=simple for a service that forks to background, or Type=forking for a service that doesn't create a PID file.
8. 8.Restart Condition Not Met: Restart=on-failure is set but the service exits with exit code 0, or RestartSec is too short causing rapid restart loops that hit rate limits.

Step 1: Check Service Status and Recent Logs

First, examine the current state of the service and what logs say.

```bash # Check if service is enabled systemctl is-enabled myapp

# enabled = starts on boot # disabled = doesn't start on boot # masked = cannot be started at all # static = no [Install] section, used as dependency

# Check service status systemctl status myapp

# Check if unit file exists systemctl cat myapp

# If file doesn't exist: ls -la /etc/systemd/system/myapp.service ls -la /lib/systemd/system/myapp.service ls -la /usr/lib/systemd/system/myapp.service

# Check recent logs journalctl -u myapp -n 50

# Check logs since last boot journalctl -u myapp -b

# Check logs from previous boot (if failed after reboot) journalctl -u myapp -b -1

# Check logs in real-time journalctl -u myapp -f

# Check systemd journal for all service failures journalctl -p err -b

# Check systemd messages dmesg | grep -i systemd | grep myapp

# List all failed services systemctl list-units --state=failed

# Check service details systemctl show myapp

# Key properties to check: systemctl show myapp -p ActiveState,SubState,ExecMainStatus,Result ```

Step 2: Examine Unit File Configuration

Review the service unit file for configuration issues.

```bash # View the complete unit file systemctl cat myapp

# Or view the source file: cat /etc/systemd/system/myapp.service

# Typical correct unit file structure: cat > /etc/systemd/system/myapp.service << 'EOF' [Unit] Description=My Application Service Documentation=https://example.com/docs After=network.target Requires=network.target Wants=network-online.target After=network-online.target

[Service] Type=simple User=myapp Group=myapp WorkingDirectory=/opt/myapp Environment="APP_ENV=production" EnvironmentFile=/etc/myapp/config.env ExecStart=/usr/bin/myapp --config /etc/myapp/config.yaml ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure RestartSec=5s TimeoutStartSec=30 TimeoutStopSec=30

# Security hardening NoNewPrivileges=true PrivateTmp=true ProtectSystem=strict ProtectHome=true ReadWritePaths=/var/lib/myapp /var/log/myapp

[Install] WantedBy=multi-user.target EOF

# Validate unit file syntax: systemd-analyze verify /etc/systemd/system/myapp.service

# Check for common issues: # 1. Missing [Install] section = cannot enable # 2. Wrong ExecStart path = command not found # 3. Missing User/Group = runs as root # 4. Wrong Type = service detection issues # 5. Missing After=network.target = network not ready

# Check if ExecStart path exists: which myapp ls -la /usr/bin/myapp

# Check if working directory exists: ls -la /opt/myapp

# Check if user exists: id myapp

# Check if config files exist: ls -la /etc/myapp/config.yaml ls -la /etc/myapp/config.env ```

Step 3: Enable Service and Verify Symlinks

Ensure the service is enabled and symlinks are created correctly.

```bash # Enable service to start on boot systemctl enable myapp

# Created symlink /etc/systemd/system/multi-user.target.wants/myapp.service → /etc/systemd/system/myapp.service.

# Verify symlink was created: ls -la /etc/systemd/system/multi-user.target.wants/myapp.service

# Check if enabled: systemctl is-enabled myapp

# If is-enabled shows "disabled" even after enable: # Check for conflicting files: ls -la /etc/systemd/system/multi-user.target.wants/ | grep myapp

# Check for masking: systemctl is-active myapp # If masked, unmask: systemctl unmask myapp

# Enable with explicit path: systemctl enable /etc/systemd/system/myapp.service

# Reload systemd daemon to pick up changes: systemctl daemon-reload

# Reset failed state if needed: systemctl reset-failed myapp

# List all enabled services: systemctl list-unit-files --type=service | grep enabled

# Check what target the service is wanted by: systemctl show myapp -p WantedBy

# Should show: WantedBy=multi-user.target

# Verify the [Install] section has WantedBy: grep -A5 "[Install]" /etc/systemd/system/myapp.service

# If [Install] section missing, add it: cat >> /etc/systemd/system/myapp.service << 'EOF'

[Install] WantedBy=multi-user.target EOF

# Then reload and enable: systemctl daemon-reload systemctl enable myapp ```

Step 4: Test Service Startup Manually

Try starting the service manually to identify errors.

```bash # Start the service systemctl start myapp

# Check status immediately systemctl status myapp

# If failed, check exit code: systemctl show myapp -p ExecMainStatus

# Common exit codes: # 0 = success # 1 = general error # 2 = misuse of shell command # 126 = cannot execute # 127 = command not found # 128+n = signal n # 130 = Ctrl+C (128+2)

# Test the ExecStart command manually: # Extract command: EXEC_START=$(systemctl cat myapp | grep ExecStart= | head -1 | sed 's/ExecStart=//') echo "Command: $EXEC_START"

# Run as the service user: sudo -u myapp /usr/bin/myapp --config /etc/myapp/config.yaml

# Check for errors

# If command runs fine manually but fails as service: # Check environment variables: systemctl show myapp -p Environment

# Add missing env vars to unit file: # Environment="PATH=/usr/local/bin:/usr/bin" # EnvironmentFile=/etc/myapp/env

# Test with debug logging: journalctl -u myapp -f & systemctl restart myapp

# Check if it's a timing issue (dependency not ready): # Add more dependencies: # After=network-online.target # Wants=network-online.target

# Check if port is already in use: ss -tlnp | grep 8080 netstat -tlnp | grep 8080

# If port in use, find what's using it: lsof -i :8080 ```

Step 5: Fix Permission and Path Issues

Resolve permission errors and missing files.

```bash # Check executable permissions: ls -la /usr/bin/myapp # Should have execute permission (-rwxr-xr-x)

# If no execute permission: chmod +x /usr/bin/myapp

# Check file ownership: ls -la /usr/bin/myapp

# If owned by wrong user: chown root:root /usr/bin/myapp

# Check working directory: ls -la /opt/myapp

# If doesn't exist: mkdir -p /opt/myapp chown myapp:myapp /opt/myapp

# Check if service user can read config: sudo -u myapp cat /etc/myapp/config.yaml

# If permission denied: chown myapp:myapp /etc/myapp/config.yaml chmod 640 /etc/myapp/config.yaml

# Check directories the service writes to: ls -la /var/lib/myapp ls -la /var/log/myapp

# Create if missing: mkdir -p /var/lib/myapp /var/log/myapp chown myapp:myapp /var/lib/myapp /var/log/myapp chmod 750 /var/lib/myapp /var/log/myapp

# Check SELinux context (if SELinux enabled): getenforce # If Enforcing, check context: ls -Z /usr/bin/myapp restorecon -v /usr/bin/myapp

# Check for SELinux denials: ausearch -m AVC -ts recent | grep myapp # If found, create policy: audit2allow -a -M myapp_policy semodule -i myapp_policy.pp

# Check AppArmor (if enabled): aa-status # If blocking, check profile: aa-logprof

# Check if port < 1024 requires capability: # Services binding to privileged ports need CAP_NET_BIND_SERVICE # Add to unit file: # AmbientCapabilities=CAP_NET_BIND_SERVICE

# Test running as service user: su - myapp -s /bin/bash -c "/usr/bin/myapp --config /etc/myapp/config.yaml" ```

Step 6: Fix Service Type and Restart Configuration

Configure correct service Type and restart behavior.

```bash # Check current Type: systemctl show myapp -p Type

# Common Type values: # simple - default, process runs in foreground # forking - process forks and parent exits # oneshot - runs once and exits # notify - process sends sd_notify when ready # dbus - waits for D-Bus name

# For daemons that fork (traditional Unix daemons): # Type=forking # PIDFile=/var/run/myapp.pid

# For modern services running in foreground: # Type=simple (default)

# For services that notify systemd when ready: # Type=notify

# Check Restart setting: systemctl show myapp -p Restart

# Restart values: # no - don't restart (default) # on-success - restart only if exit code is 0 # on-failure - restart if exit code != 0 # on-abnormal - restart on signal or timeout # on-watchdog - restart on watchdog timeout # on-abort - restart on uncaught signal # always - always restart

# For critical services, use: # Restart=on-failure # RestartSec=5s

# Fix rapid restart loops: # Add restart delay and limits: # Restart=on-failure # RestartSec=5s # StartLimitIntervalSec=60s # StartLimitBurst=3

# Check current restart settings: systemctl show myapp -p Restart,RestartSec,StartLimitIntervalSec,StartLimitBurst

# Update unit file: sed -i 's/^Restart=.*/Restart=on-failure/' /etc/systemd/system/myapp.service sed -i 's/^RestartSec=.*/RestartSec=5s/' /etc/systemd/system/myapp.service

# Or add if missing: sed -i '/[Service]/a Restart=on-failure\nRestartSec=5s' /etc/systemd/system/myapp.service

# Reload and restart: systemctl daemon-reload systemctl restart myapp ```

Step 7: Fix Dependency Ordering

Ensure service starts after its dependencies are ready.

```bash # Check current dependencies: systemctl list-dependencies myapp

# Check reverse dependencies (what depends on this): systemctl list-dependencies --reverse myapp

# Check ordering: systemctl show myapp -p After,Before,Wants,Requires

# Common dependencies: # network.target - network stack initialized # network-online.target - network connectivity available # remote-fs.target - remote filesystems mounted # local-fs.target - local filesystems mounted # nss-lookup.target - name resolution available # time-sync.target - time synchronized

# For network-dependent services: # After=network-online.target # Wants=network-online.target

# For database-dependent services: # After=network.target postgresql.service # Requires=postgresql.service

# For filesystem-dependent services: # After=remote-fs.target # Requires=remote-fs.target

# Check if dependency exists: systemctl list-unit-files | grep postgresql

# Test if service waits for dependencies: systemd-analyze plot > boot_analysis.svg # View in browser to see boot sequence

# Check boot order: systemd-analyze critical-chain myapp

# Check what's slowing boot: systemd-analyze blame | head -20

# Add missing dependencies: sed -i '/[Unit]/a After=network-online.target postgresql.service\nRequires=postgresql.service' /etc/systemd/system/myapp.service

# Reload and test: systemctl daemon-reload systemctl restart myapp ```

Step 8: Test Service Startup on Boot

Verify service starts correctly after reboot.

```bash # Before rebooting, check service is enabled: systemctl is-enabled myapp

# Check last exit status: systemctl show myapp -p ExecMainStatus

# Save current status: systemctl status myapp > /tmp/pre-reboot-status.txt

# Reboot: reboot

# After reboot, check status: systemctl status myapp systemctl is-active myapp

# Check if started automatically: journalctl -u myapp -b | head -20

# Check boot time: systemd-analyze time

# Check service startup time: systemd-analyze blame | grep myapp

# If service didn't start: # Check if enabled: systemctl is-enabled myapp

# Check logs for why it failed: journalctl -u myapp -b

# Check if dependency failed: journalctl -u network-online.target -b journalctl -u postgresql -b

# Test with explicit start after boot: systemctl start myapp

# If manual start works but auto start fails: # Check for race condition with dependencies # Add more conservative After= settings

# Simulate boot order: systemd-analyze verify /etc/systemd/system/myapp.service ```

Step 9: Set Up Monitoring and Alerting

Create monitoring to detect service failures.

```bash # Create service monitoring script: cat > /usr/local/bin/check-service.sh << 'EOF' #!/bin/bash # Systemd Service Health Check

SERVICE="${1:-myapp}" ALERT_EMAIL="${ALERT_EMAIL:-ops@company.com}" LOG_FILE="/var/log/service-monitor.log"

# Check if service is active IS_ACTIVE=$(systemctl is-active $SERVICE)

if [ "$IS_ACTIVE" != "active" ]; then echo "$(date): $SERVICE is not active (status: $IS_ACTIVE)" >> $LOG_FILE

# Get recent logs RECENT_LOGS=$(journalctl -u $SERVICE -n 20 --no-pager)

# Try to restart echo "$(date): Attempting to restart $SERVICE" >> $LOG_FILE systemctl restart $SERVICE

sleep 5

# Check if restart worked NEW_STATUS=$(systemctl is-active $SERVICE) if [ "$NEW_STATUS" = "active" ]; then echo "$(date): $SERVICE restarted successfully" >> $LOG_FILE echo -e "Service $SERVICE was down and has been restarted.\n\nPrevious status: $IS_ACTIVE\nCurrent status: $NEW_STATUS\n\nRecent logs:\n$RECENT_LOGS" | mail -s "Service Restarted: $SERVICE" $ALERT_EMAIL exit 0 else echo "$(date): Failed to restart $SERVICE" >> $LOG_FILE echo -e "Service $SERVICE is down and automatic restart failed.\n\nStatus: $NEW_STATUS\n\nRecent logs:\n$RECENT_LOGS" | mail -s "SERVICE DOWN: $SERVICE" $ALERT_EMAIL exit 1 fi fi

# Check if service is enabled IS_ENABLED=$(systemctl is-enabled $SERVICE 2>/dev/null) if [ "$IS_ENABLED" != "enabled" ]; then echo "$(date): WARNING - $SERVICE is not enabled for auto-start" >> $LOG_FILE fi

# Check for recent restarts (might indicate crash loop) RESTART_COUNT=$(journalctl -u $SERVICE --since "10 minutes ago" | grep -c "Started|Starting") if [ "$RESTART_COUNT" -gt 3 ]; then echo "$(date): WARNING - $SERVICE has restarted $RESTART_COUNT times in last 10 minutes" >> $LOG_FILE fi

echo "$(date): $SERVICE is healthy" >> $LOG_FILE exit 0 EOF

chmod +x /usr/local/bin/check-service.sh

# Test the check: /usr/local/bin/check-service.sh myapp

# Add to cron: crontab -e # Add: */2 * * * * /usr/local/bin/check-service.sh myapp

# Create systemd watchdog script: cat > /usr/local/bin/service-watchdog.sh << 'EOF' #!/bin/bash # Systemd watchdog notification

SERVICE="myapp" TIMEOUT=60

while true; do if systemctl is-active --quiet $SERVICE; then # Send watchdog notification systemd-notify WATCHDOG=1 fi sleep $((TIMEOUT / 2)) done EOF

chmod +x /usr/local/bin/service-watchdog.sh

# Add to service unit: # WatchdogSec=60 # ExecStartPost=/usr/local/bin/service-watchdog.sh

# Create Prometheus node exporter textfile: cat > /var/lib/node_exporter/textfile_collector/systemd_services.prom << 'EOF' # HELP systemd_service_active Whether service is active # TYPE systemd_service_active gauge systemd_service_active{service="myapp"} 1 EOF

# Update with actual status: systemctl is-active myapp >/dev/null && echo 'systemd_service_active{service="myapp"} 1' || echo 'systemd_service_active{service="myapp"} 0' ```

Step 10: Document and Standardize Service Deployment

Create templates and procedures for consistent service deployment.

```bash # Create service template: cat > /etc/systemd/system/template.service << 'EOF' [Unit] Description=TEMPLATE_SERVICE_NAME Service Documentation=https://example.com/docs/TEMPLATE_SERVICE_NAME After=network.target network-online.target Wants=network-online.target

[Service] Type=simple User=TEMPLATE_USER Group=TEMPLATE_GROUP WorkingDirectory=TEMPLATE_WORKDIR

# Environment Environment="PATH=/usr/local/bin:/usr/bin:/bin" EnvironmentFile=-/etc/default/TEMPLATE_SERVICE_NAME

# Executable ExecStart=TEMPLATE_EXEC_START ExecReload=/bin/kill -HUP $MAINPID

# Restart behavior Restart=on-failure RestartSec=5s TimeoutStartSec=30 TimeoutStopSec=30

# Limits LimitNOFILE=65535 LimitNPROC=4096

# Security NoNewPrivileges=true PrivateTmp=true ProtectSystem=strict ProtectHome=true ReadWritePaths=TEMPLATE_DATA_DIR TEMPLATE_LOG_DIR

[Install] WantedBy=multi-user.target EOF

# Create deployment script: cat > /usr/local/bin/deploy-service.sh << 'EOF' #!/bin/bash set -e

SERVICE_NAME="${1:-}" SERVICE_USER="${2:-$SERVICE_NAME}" SERVICE_GROUP="${3:-$SERVICE_NAME}" EXEC_START="${4:-}" WORKDIR="${5:-/opt/$SERVICE_NAME}"

if [ -z "$SERVICE_NAME" ] || [ -z "$EXEC_START" ]; then echo "Usage: $0 service_name [user] [group] exec_start [workdir]" exit 1 fi

UNIT_FILE="/etc/systemd/system/${SERVICE_NAME}.service"

echo "Creating service: $SERVICE_NAME"

# Create user if needed if ! id $SERVICE_USER &>/dev/null; then useradd -r -s /bin/false $SERVICE_USER echo "Created user: $SERVICE_USER" fi

# Create directories mkdir -p $WORKDIR mkdir -p /var/log/$SERVICE_NAME mkdir -p /var/lib/$SERVICE_NAME

# Create unit file cat > $UNIT_FILE << SERVICEEOF [Unit] Description=$SERVICE_NAME Service After=network.target

[Service] Type=simple User=$SERVICE_USER Group=$SERVICE_GROUP WorkingDirectory=$WORKDIR ExecStart=$EXEC_START Restart=on-failure RestartSec=5s

[Install] WantedBy=multi-user.target SERVICEEOF

# Set permissions chown -R $SERVICE_USER:$SERVICE_GROUP $WORKDIR chown -R $SERVICE_USER:$SERVICE_GROUP /var/log/$SERVICE_NAME chown -R $SERVICE_USER:$SERVICE_GROUP /var/lib/$SERVICE_NAME

# Enable and start systemctl daemon-reload systemctl enable $SERVICE_NAME systemctl start $SERVICE_NAME

echo "Service $SERVICE_NAME deployed and started" systemctl status $SERVICE_NAME EOF

chmod +x /usr/local/bin/deploy-service.sh

# Usage: # /usr/local/bin/deploy-service.sh myapp myapp myapp "/usr/bin/myapp --config /etc/myapp/config.yaml" /opt/myapp ```

Checklist for Fixing Systemd Service Startup Issues

Verify the Fix

After fixing systemd service issues, verify everything works:

```bash # 1. Service is enabled systemctl is-enabled myapp # Should show: enabled

# 2. Service is active systemctl is-active myapp # Should show: active

# 3. Service starts on boot reboot # After reboot: systemctl is-active myapp # Should show: active

# 4. Service restarts on failure systemctl kill -s SIGKILL myapp sleep 10 systemctl is-active myapp # Should show: active (restarted)

# 5. No errors in logs journalctl -u myapp -n 20 # Should show successful startup

# 6. Dependencies are correct systemctl list-dependencies myapp # Should show required services

# 7. Unit file syntax is valid systemd-analyze verify /etc/systemd/system/myapp.service # Should show no errors

# 8. Service responds correctly curl http://localhost:8080/health # Should get response

# 9. Monitoring check passes /usr/local/bin/check-service.sh myapp # Exit code 0

# 10. Service survives system updates # After apt upgrade or similar: systemctl status myapp # Still active ```

Related Issues

• [Fix Systemd Service Restart Loop](/articles/fix-systemd-service-restart-loop) - Continuous restart issues
• [Fix Linux Service Permission Denied](/articles/fix-linux-service-permission-denied) - Permission errors
• [Fix Systemd Timer Not Running](/articles/fix-systemd-timer-not-running) - Scheduled task issues
• [Fix Process Zombie Linux](/articles/fix-process-zombie-linux) - Zombie process cleanup
• [Fix Linux Out of Memory OOM Kill](/articles/fix-linux-out-of-memory-oom-kill) - OOM issues
• [Fix Systemd Service Timeout](/articles/fix-systemd-service-timeout) - Service timeout issues
• [Fix Linux Port Already in Use](/articles/fix-linux-port-already-in-use) - Port binding conflicts