Introduction

In async Rust, holding a std::sync::Mutex guard across an .await point causes the Tokio runtime to deadlock. The synchronous mutex blocks the entire thread when contended, but async tasks can be moved between threads. When a guard is held across an await, the task yields, the guard remains locked, and other tasks on the same thread cannot acquire it. This is one of the most common async Rust mistakes in production.

Symptoms

  • Tokio runtime hangs completely, no tasks make progress
  • No panic or error message, the application just stops responding
  • Health check endpoint stops responding
  • tokio-console shows tasks stuck waiting on mutex
  • Works in single-threaded runtime but deadlocks with multi-threaded

Debug with tokio-console: ``toml # Cargo.toml [dependencies] tokio = { version = "1", features = ["full", "tracing"] } console-subscriber = "0.2"

rust
// main.rs - enable tokio-console
#[tokio::main]
async fn main() {
    console_subscriber::init();
    // Run your app
}
// Then run: RUSTFLAGS="--cfg tokio_unstable" cargo run
// And: tokio-console

Common Causes

  • std::sync::Mutex guard held across .await point
  • Lock contention on synchronous mutex in multi-threaded runtime
  • Mutex acquired in one async task, awaited, then released
  • Using Mutex inside Arc without async-aware version
  • Nested lock acquisition creating classic deadlock pattern

Step-by-Step Fix

  1. 1.Replace std::sync::Mutex with tokio::sync::Mutex:
  2. 2.```rust
  3. 3.use std::sync::Arc;
  4. 4.use tokio::sync::Mutex;

// WRONG - std::sync::Mutex blocks thread across await use std::sync::Mutex as StdMutex; let data: Arc<StdMutex<Vec<String>>> = Arc::new(StdMutex::new(vec![]));

tokio::spawn(async move { let guard = data.lock().unwrap(); // Lock acquired tokio::time::sleep(Duration::from_secs(1)).await; // DEADLOCK: guard held across await guard.push("item".to_string()); });

// CORRECT - tokio::sync::Mutex yields properly let data: Arc<Mutex<Vec<String>>> = Arc::new(Mutex::new(vec![]));

tokio::spawn(async move { let guard = data.lock().await; // Async lock - yields if contended tokio::time::sleep(Duration::from_secs(1)).await; // Safe: lock releases when task yields // Note: tokio MutexGuard is NOT Send, so it cannot be held across await to different task drop(guard); // Explicitly drop before await tokio::time::sleep(Duration::from_secs(1)).await; let guard = data.lock().await; guard.push("item".to_string()); }); ```

  1. 1.Minimize lock scope to avoid holding across await:
  2. 2.```rust
  3. 3.use std::sync::{Arc, Mutex};

let cache: Arc<Mutex<HashMap<String, String>>> = Arc::new(Mutex::new(HashMap::new()));

// WRONG - lock held during expensive operation async fn get_or_insert(cache: &Arc<Mutex<HashMap<String, String>>>, key: &str) -> String { let mut map = cache.lock().unwrap(); if let Some(val) = map.get(key) { return val.clone(); } let value = expensive_computation(key).await; // Lock held during await! map.insert(key.to_string(), value.clone()); value }

// CORRECT - lock only for read, then compute, then lock for write async fn get_or_insert_fixed(cache: &Arc<Mutex<HashMap<String, String>>>, key: &str) -> String { // Check cache first with short lock { let map = cache.lock().unwrap(); if let Some(val) = map.get(key) { return val.clone(); } } // Lock released here

// Compute without holding lock let value = expensive_computation(key).await;

// Lock briefly to insert { let mut map = cache.lock().unwrap(); // Double-check in case another task inserted while we computed if let Some(val) = map.get(key) { return val.clone(); } map.insert(key.to_string(), value.clone()); }

value } ```

  1. 1.Use RwLock for read-heavy workloads:
  2. 2.```rust
  3. 3.use tokio::sync::RwLock;

let config: Arc<RwLock<AppConfig>> = Arc::new(RwLock::new(AppConfig::default()));

// Multiple readers can access simultaneously let reader1 = config.read().await; let reader2 = config.read().await; // Both can read at the same time

// Writer requires exclusive access let mut writer = config.write().await; // Blocks all readers writer.max_retries = 10; ```

  1. 1.Detect potential deadlocks at runtime:
  2. 2.```rust
  3. 3.use parking_lot::Mutex; // parking_lot has deadlock detection in debug mode

// Enable deadlock detection #[cfg(debug_assertions)] parking_lot::deadlock::enable_deadlock_detection();

// Check for deadlocks periodically fn spawn_deadlock_checker() { use std::time::Duration;

std::thread::spawn(|| { loop { std::thread::sleep(Duration::from_secs(10)); let deadlocks = parking_lot::deadlock::check_deadlock(); if deadlocks.is_empty() { continue; }

eprintln!("{} deadlocks detected", deadlocks.len()); for (i, threads) in deadlocks.iter().enumerate() { eprintln!("Deadlock #{}", i); for t in threads { eprintln!(" Thread ID: {:?}", t.thread_id()); eprintln!(" Backtrace: {:?}", t.backtrace()); } } } }); } ```

Prevention

  • Use clippy::await_holding_lock lint to catch sync Mutex across await
  • Prefer tokio::sync::Mutex when the lock must be held across await points
  • Keep critical sections as short as possible
  • Use parking_lot::Mutex for better performance than std::sync::Mutex
  • Enable tokio-console in staging to detect task starvation
  • Write integration tests that exercise concurrent access patterns