Introduction

The Polly Bulkhead policy limits the number of concurrent executions of a given operation, protecting resources from exhaustion. It works like a ship's bulkhead — when one compartment fills, the damage is contained. The policy uses a semaphore to control concurrency and a queue for excess requests. When the bulkhead is misconfigured, too many requests get through causing resource exhaustion, or legitimate requests are rejected with BulkheadRejectedException because the queue is full. With Polly 8+ and the Polly.Core library, bulkhead is replaced by ConcurrencyLimiter in resilience pipelines.

Symptoms

  • BulkheadRejectedException thrown even under low load
  • Too many concurrent calls overwhelm downstream service
  • Bulkhead maxParallelization limit not enforced
  • Queued actions timeout before execution
  • Resource exhaustion despite bulkhead policy being configured
  • BulkheadRejection causes cascading failures in caller

Error output: `` Polly.Bulkhead.BulkheadRejectedException: Too many executions have occurred within the bulkhead policy limit of 10.

Common Causes

  • maxParallelization set too high for downstream capacity
  • maxQueuingActions set to 0, causing immediate rejection
  • Same bulkhead instance shared across unrelated operations
  • Bulkhead not combined with timeout policy, causing queue buildup
  • Async operations not properly awaited, holding bulkhead slots
  • Queue wait timeout too short for actual operation duration

Step-by-Step Fix

  1. 1.Configure bulkhead with correct parallelization and queue:
  2. 2.```csharp
  3. 3.using Polly;

// Legacy Polly (v7) var bulkhead = Policy.BulkheadAsync( maxParallelization: 10, // Max concurrent executions maxQueuingActions: 50, // Max requests waiting in queue onBulkheadRejectedAsync: async context => { // Log when requests are rejected var logger = context.GetLogger(); logger.LogWarning("Bulkhead full - request rejected. Endpoint: {Endpoint}", context.OperationKey); });

// Usage try { var result = await bulkhead.ExecuteAsync(async () => { return await _httpClient.GetAsync("/api/data"); }); } catch (BulkheadRejectedException ex) { // Return 429 Too Many Requests or fallback return Results.StatusCode(429); }

// Per-endpoint bulkheads for isolation var userBulkhead = Policy.BulkheadAsync(maxParallelization: 20, maxQueuingActions: 100); var orderBulkhead = Policy.BulkheadAsync(maxParallelization: 5, maxQueuingActions: 20); var reportBulkhead = Policy.BulkheadAsync(maxParallelization: 2, maxQueuingActions: 5);

// Each endpoint has its own resource pool app.MapGet("/api/users", async () => await userBulkhead.ExecuteAsync(GetUsers)); app.MapGet("/api/orders", async () => await orderBulkhead.ExecuteAsync(GetOrders)); app.MapGet("/api/reports", async () => await reportBulkhead.ExecuteAsync(GenerateReport)); ```

  1. 1.Combine bulkhead with timeout and retry for robust isolation:
  2. 2.```csharp
  3. 3.var bulkheadTimeoutRetry = Policy
  4. 4..BulkheadAsync(
  5. 5.maxParallelization: 10,
  6. 6.maxQueuingActions: 50,
  7. 7.onBulkheadRejectedAsync: async ctx =>
  8. 8.{
  9. 9.// Track rejection metrics
  10. 10.Metrics.IncrementCounter("bulkhead.rejections");
  11. 11.})
  12. 12..WrapAsync(
  13. 13.Policy.TimeoutAsync(
  14. 14.TimeSpan.FromSeconds(30),
  15. 15.TimeoutStrategy.Pessimistic,
  16. 16.async (context, timespan, task) =>
  17. 17.{
  18. 18.var logger = context.GetLogger();
  19. 19.logger.LogWarning("Operation timed out after {Timeout}s", timespan.TotalSeconds);
  20. 20.}))
  21. 21..WrapAsync(
  22. 22.Policy.Handle<HttpRequestException>()
  23. 23..OrResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
  24. 24..WaitAndRetryAsync(
  25. 25.retryCount: 3,
  26. 26.sleepDurationProvider: attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)),
  27. 27.onRetryAsync: (outcome, timespan, attempt, context) =>
  28. 28.{
  29. 29.var logger = context.GetLogger();
  30. 30.logger.LogWarning("Retry {Attempt} after {Delay}s", attempt, timespan.TotalSeconds);
  31. 31.return Task.CompletedTask;
  32. 32.}));

// Usage - policies execute in order: Retry -> Timeout -> Bulkhead -> Action var result = await bulkheadTimeoutRetry.ExecuteAsync(async () => { return await _httpClient.GetAsync("/api/data"); }); ```

  1. 1.Use Polly 8+ ConcurrencyLimiter in resilience pipeline:
  2. 2.```csharp
  3. 3.// Polly 8+ (Polly.Core) - new resilience pipeline approach
  4. 4.using Polly.RateLimiting;

var resiliencePipeline = new ResiliencePipelineBuilder() .AddConcurrencyLimiter(new ConcurrencyLimiterOptions { PermitLimit = 10, // Max concurrent executions QueueLimit = 50, // Max queued requests QueueTimeout = TimeSpan.FromSeconds(60), // Time to wait in queue }) .AddTimeout(TimeSpan.FromSeconds(30)) .AddRetry(new RetryStrategyOptions<HttpResponseMessage> { MaxRetryAttempts = 3, BackoffType = DelayBackoffType.Exponential, ShouldHandle = new PredicateBuilder<HttpResponseMessage>() .Handle<HttpRequestException>() .HandleResult(r => !r.IsSuccessStatusCode), Delay = TimeSpan.FromSeconds(1), }) .Build();

// Monitor concurrency metrics var pipeline = new ResiliencePipelineBuilder() .AddConcurrencyLimiter(new ConcurrencyLimiterOptions { PermitLimit = 10, QueueLimit = 50, QueueTimeout = TimeSpan.FromSeconds(60), }) .Build();

// Track active executions var activeCount = pipeline.TelemetryUtil.GetMetric("active-executions"); Console.WriteLine($"Active executions: {activeCount}"); ```

  1. 1.Monitor bulkhead health and capacity:
  2. 2.```csharp
  3. 3.// Track bulkhead state for diagnostics
  4. 4.public class BulkheadMonitor
  5. 5.{
  6. 6.private readonly SemaphoreSlim _semaphore;
  7. 7.private readonly int _maxParallelization;
  8. 8.private readonly int _maxQueuingActions;
  9. 9.private int _currentExecutions;
  10. 10.private int _currentQueue;

public BulkheadMonitor(int maxParallelization, int maxQueuingActions) { _maxParallelization = maxParallelization; _maxQueuingActions = maxQueuingActions; _semaphore = new SemaphoreSlim(maxParallelization, maxParallelization); }

public async Task<T> ExecuteAsync<T>(Func<Task<T>> action) { var queued = Interlocked.Increment(ref _currentQueue); if (queued > _maxQueuingActions + _maxParallelization) { Interlocked.Decrement(ref _currentQueue); throw new BulkheadRejectedException( $"Bulkhead full: {queued} pending, max {_maxQueuingActions + _maxParallelization}"); }

try { await _semaphore.WaitAsync(); Interlocked.Decrement(ref _currentQueue); Interlocked.Increment(ref _currentExecutions);

try { return await action(); } finally { Interlocked.Decrement(ref _currentExecutions); _semaphore.Release(); } } catch { Interlocked.Decrement(ref _currentQueue); throw; } }

public string GetStatus() => $"Executions: {_currentExecutions}/{_maxParallelization}, " + $"Queued: {_currentQueue}/{_maxQueuingActions}"; } ```

Prevention

  • Size maxParallelization based on downstream service capacity, not available threads
  • Set maxQueuingActions to absorb short bursts without rejecting requests
  • Always combine bulkhead with a timeout policy to prevent queue starvation
  • Use separate bulkhead instances for different operations with different resource profiles
  • Monitor bulkhead rejection rates and adjust limits based on production metrics
  • Use Polly 8+ ConcurrencyLimiter for new projects — it integrates with the resilience pipeline builder
  • Add health check endpoints that expose bulkhead utilization
  • Set QueueTimeout to prevent requests from waiting indefinitely in the queue