Introduction

BackgroundService is the base class for long-running background tasks in .NET. If an unhandled exception escapes the ExecuteAsync method, it propagates to the host and crashes the entire application process. Unlike web requests where exceptions are caught by the middleware pipeline, background service exceptions are fatal by default.

Symptoms

  • Entire application crashes when a background task encounters an error
  • Unhandled exception. System.Exception in logs followed by process exit
  • Other services in the same host stop running
  • Docker container restarts repeatedly due to background task crash
  • Exception in ExecuteAsync not caught by global exception handler

Example crash: `` Unhandled exception. System.InvalidOperationException: Queue connection lost and cannot be recovered. at MyApp.WorkerService.ExecuteAsync(CancellationToken stoppingToken) at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token) at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>g__LogAndRethrow|15_3(<>c__DisplayClass15_0&)

Common Causes

  • No try-catch around the main loop in ExecuteAsync
  • Network connection failures not handled in message processing loops
  • Database connection lost during long-running processing
  • External API permanently unavailable
  • CancellationToken not checked, causing work after cancellation

Step-by-Step Fix

  1. 1.Wrap ExecuteAsync in try-catch with logging:
  2. 2.```csharp
  3. 3.public class DataProcessingService : BackgroundService
  4. 4.{
  5. 5.private readonly ILogger<DataProcessingService> _logger;

public DataProcessingService(ILogger<DataProcessingService> logger) { _logger = logger; }

protected override async Task ExecuteAsync(CancellationToken stoppingToken) { try { await ProcessLoopAsync(stoppingToken); } catch (OperationCanceledException) { _logger.LogInformation("Service stopped gracefully"); } catch (Exception ex) { _logger.LogCritical(ex, "Background service crashed"); // Optionally: attempt restart or trigger alert throw; // Re-throw to let the host decide } }

private async Task ProcessLoopAsync(CancellationToken stoppingToken) { while (!stoppingToken.IsCancellationRequested) { try { await ProcessNextItemAsync(stoppingToken); } catch (Exception ex) { _logger.LogError(ex, "Error processing item, continuing..."); await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken); } } } } ```

  1. 1.Implement retry with backoff for transient failures:
  2. 2.```csharp
  3. 3.protected override async Task ExecuteAsync(CancellationToken stoppingToken)
  4. 4.{
  5. 5.var retryPolicy = Policy
  6. 6..Handle<SqlException>(ex => ex.Number == -2 || ex.Number == 53) // Timeout or network
  7. 7..Or<HttpRequestException>()
  8. 8..WaitAndRetryForeverAsync(
  9. 9.retryAttempt => TimeSpan.FromSeconds(Math.Min(Math.Pow(2, retryAttempt), 60)),
  10. 10.(exception, timespan, retryCount, context) =>
  11. 11.{
  12. 12._logger.LogWarning(exception,
  13. 13."Transient failure, retrying in {Delay} seconds (attempt {Retry})",
  14. 14.timespan.TotalSeconds, retryCount);
  15. 15.});

await retryPolicy.ExecuteAsync(async (ct) => { while (!ct.IsCancellationRequested) { await ProcessItemsAsync(ct); await Task.Delay(TimeSpan.FromSeconds(1), ct); } }, stoppingToken); } ```

  1. 1.Configure host to not crash on background service failure:
  2. 2.```csharp
  3. 3.// .NET 8+ option: background services do not crash the host
  4. 4.builder.Services.AddHostedService<DataProcessingService>();

// Configure the host to handle background service failures gracefully builder.Host.UseDefaultServiceProvider(options => { options.ValidateScopes = builder.Environment.IsDevelopment(); options.ValidateOnBuild = true; }); ```

  1. 1.Monitor and alert on background service health:
  2. 2.```csharp
  3. 3.public class MonitoredBackgroundService : BackgroundService
  4. 4.{
  5. 5.private readonly IMetricsCollector _metrics;
  6. 6.private DateTime _lastSuccessfulProcessing = DateTime.UtcNow;

protected override async Task ExecuteAsync(CancellationToken stoppingToken) { while (!stoppingToken.IsCancellationRequested) { try { var processed = await ProcessItemsAsync(stoppingToken); _lastSuccessfulProcessing = DateTime.UtcNow; _metrics.Increment("items.processed", processed); } catch (Exception ex) { _metrics.Increment("items.processing.errors"); _logger.LogError(ex, "Processing error"); await Task.Delay(TimeSpan.FromSeconds(10), stoppingToken); } } } } ```

Prevention

  • Always wrap ExecuteAsync in a try-catch at the top level
  • Use Polly for retry policies on transient failures
  • Check stoppingToken.IsCancellationRequested in long loops
  • Add health checks that verify the background service is still processing
  • Monitor the time since last successful processing
  • Use IHostApplicationLifetime.StopApplication() for unrecoverable errors
  • Configure systemd/Docker to restart the process on crash