How to Fix Linux at Job Failed Errors

The at command schedules one-time tasks for future execution, but when these jobs fail to run, diagnosing the issue can be tricky. Unlike cron's recurring schedule, at jobs are fire-and-forget, making failures harder to detect.

Understanding the at Command

The at daemon (atd) manages scheduled one-time jobs. Jobs are stored in a queue and executed at the specified time. Unlike cron, at jobs are removed after execution.

Typical Error Messages and Symptoms

Cannot open lock file /var/spool/cron/atjobs/.SEQ: Permission denied
You do not have permission to use at.
Warning: commands will be executed using /bin/sh
Can't open /var/run/atd.pid to signal atd. No atd running?

Common symptoms: - Jobs don't execute at scheduled time - Jobs disappear without running - Permission denied when scheduling - at daemon not running

Diagnosing at Job Issues

Step 1: Check if at Daemon is Running

```bash # Check atd service status systemctl status atd

# Check if process exists ps aux | grep atd pgrep -a atd

# Start atd if not running systemctl start atd systemctl enable atd

# If systemd unit doesn't exist atd ```

Step 2: Check at Job Queue

```bash # List pending jobs for current user atq

# List all jobs (as root) sudo atq

# View job details at -c job_number

# View job with less paging at -c job_number | less

# Check job files in spool directory ls -la /var/spool/cron/atjobs/ ls -la /var/spool/at/jobs/ # Alternative location ```

Step 3: Check Permissions

```bash # Check at.allow and at.deny files cat /etc/at.allow # If exists, only these users can use at cat /etc/at.deny # If exists, these users cannot use at

# If neither exists, typically all users can use at # Check directory permissions ls -la /var/spool/cron/atjobs/ ls -la /var/spool/at/

# Directory should be owned by daemon or root # Files should be owned by user who created them ```

Step 4: Check Logs

```bash # Check system logs for at daemon messages journalctl -u atd -n 50

# Check syslog for at messages grep -i "at[" /var/log/syslog grep -i "atd" /var/log/messages

# Check mail for at job output (default behavior) mail # Or check user's mail file cat /var/mail/$USER ```

Common Issues and Solutions

Issue 1: at Daemon Not Running

```bash # Check if atd is installed which atd dpkg -l | grep at # Debian/Ubuntu rpm -qa | grep at # RHEL/CentOS

# Install if missing sudo apt-get install at # Debian/Ubuntu sudo yum install at # RHEL/CentOS

# Start the service sudo systemctl start atd sudo systemctl enable atd

# Verify it's running systemctl status atd pgrep -a atd ```

Issue 2: Permission Denied

```bash # Check if user is allowed cat /etc/at.allow cat /etc/at.deny

# If at.allow exists, add your user echo "$USER" | sudo tee -a /etc/at.allow

# If at.deny exists and user is in it, remove them sudo sed -i "/^$USER$/d" /etc/at.deny

# Check spool directory permissions ls -la /var/spool/cron/atjobs/

# Fix permissions if needed sudo chmod 1770 /var/spool/cron/atjobs/ sudo chown daemon:daemon /var/spool/cron/atjobs/ # Or root:root depending on distro ```

Issue 3: Jobs Not Executing

```bash # Verify job exists in queue atq

# Check job content at -c 1 # Where 1 is the job number

# Check job execution time atq -q a # Show jobs in queue 'a'

# Ensure atd is running systemctl status atd

# Check if job already executed grep "at[" /var/log/syslog | tail -20

# Job output is mailed by default, check mail cat /var/mail/$USER

# Or redirect output when creating job echo "/path/to/script.sh > /tmp/at-output.log 2>&1" | at now + 5 minutes ```

Issue 4: Environment Variables Missing

Like cron, at jobs run with a limited environment.

```bash # When creating an at job, source your environment at now + 5 minutes << 'EOF' source /home/user/.bashrc /path/to/script.sh EOF

# Or use full paths echo "/usr/bin/python3 /home/user/script.py" | at 14:30

# Check what environment the job sees at now + 1 minute << 'EOF' env > /tmp/at-env.txt EOF # Check output after job runs cat /tmp/at-env.txt ```

Issue 5: Script Works Manually But Not in at

```bash # Use the -f flag to run a script file at -f /path/to/script.sh now + 5 minutes

# Test with a simple command first echo "date > /tmp/at-test.txt" | at now + 1 minute

# Check if it ran cat /tmp/at-test.txt

# If using shell-specific features, specify the shell # at uses /bin/sh by default echo "/bin/bash /path/to/script.sh" | at 15:00

# Or change SHELL for the job SHELL=/bin/bash at now + 5 minutes << 'EOF' your bash-specific commands here EOF ```

Issue 6: Job Queue Issues

```bash # at uses different queues (a-z, A-Z) # Lowercase = batch jobs (nice value) # Uppercase = special treatment

# Check job in specific queue atq -q a

# Submit job to specific queue echo "/path/to/script.sh" | at -q b now + 1 hour

# Batch queue runs when system load is low batch << 'EOF' /path/to/cpu-intensive-script.sh EOF

# Check batch queue atq -q = # Batch queue is '=' ```

Issue 7: Wrong Execution Time

```bash # at uses 24-hour format by default at 14:30 # 2:30 PM at 2:30 PM # Also 2:30 PM (if AM/PM is supported) at 1430 # Also 2:30 PM

# Common time specifications at now # Immediately at now + 5 minutes at now + 1 hour at now + 2 days at 14:30 tomorrow at 14:30 next week at 14:30 Feb 28 at 14:30 2/28 at teatime # 4:00 PM (some systems) at midnight # 12:00 AM at noon # 12:00 PM

# Check system timezone timedatectl date

# at uses system timezone # To verify job time atq # Shows: 1 Fri Feb 16 14:30:00 2024 a username ```

Issue 8: Output Lost

By default, at jobs mail their output to the user.

```bash # Redirect output to a file at now + 5 minutes << 'EOF' /path/to/script.sh > /tmp/job-output.log 2>&1 EOF

# Capture both stdout and stderr echo "/path/to/script.sh >> /var/log/at-job.log 2>&1" | at now + 10 minutes

# Check mail configuration which mail dpkg -l mailutils # Debian/Ubuntu

# If mail isn't configured, redirect to file # Jobs will fail silently if output can't be mailed ```

Testing and Debugging

Test at Functionality

```bash # Simple test job echo "echo 'at works at $(date)' > /tmp/at-test-\$(date +%s).txt" | at now + 1 minute

# Wait and check sleep 70 ls -la /tmp/at-test-*.txt cat /tmp/at-test-*.txt

# Test with a more complex command at now + 2 minutes << 'EOF' echo "Environment:" env > /tmp/at-environment.txt echo "Current directory: $(pwd)" >> /tmp/at-environment.txt echo "User: $(whoami)" >> /tmp/at-environment.txt echo "Path: $PATH" >> /tmp/at-environment.txt EOF

# Check output cat /tmp/at-environment.txt ```

Monitor at Jobs

```bash # Watch at queue watch -n 1 atq

# Follow at logs journalctl -u atd -f

# Create monitoring alias alias atwatch='watch -n 1 "atq && echo && sudo tail -5 /var/log/syslog | grep at"' ```

Removing and Managing Jobs

```bash # Remove a job atrm 2 # Remove job number 2

# Remove multiple jobs atrm 3 4 5

# Remove all your jobs atrm $(atq | awk '{print $1}')

# Show job content before removing at -c 2 atrm 2 ```

at vs cron: When to Use Each

Use at for: - Running a backup after hours - Sending a reminder - Executing a long job during off-peak - Any one-time delayed task

Use cron for: - Daily backups - Hourly log rotation - Weekly reports - Any recurring task

Verification

After fixing issues:

```bash # Verify atd is running systemctl status atd

# Test a simple job echo "date > /tmp/at-verify.txt" | at now + 1 minute

# Wait and check sleep 70 cat /tmp/at-verify.txt

# Check job queue is being processed atq

# Check logs journalctl -u atd -n 10 ```

Best Practices

```bash # 1. Always redirect output echo "/path/to/script.sh >> /var/log/at.log 2>&1" | at now + 1 hour

# 2. Use full paths echo "/usr/bin/rsync -av /src/ /dest/" | at midnight

# 3. Use here-documents for complex commands at now + 1 hour << 'EOF' source ~/.bashrc cd /home/user/project ./run-backup.sh >> /var/log/backup.log 2>&1 EOF

# 4. Verify job was created echo "Job created at $(date)" echo "sleep 300 && /path/to/script.sh" | at now + 5 minutes atq

# 5. Document why the job was created echo "# Post-deployment cleanup" | at now + 30 minutes

# 6. Use batch for CPU-intensive jobs batch << 'EOF' /usr/bin/nice -n 19 /path/to/cpu-intensive-job.sh EOF ```