Introduction

MongoDB $lookup aggregation memory exceeded when joining large collections. This guide provides step-by-step diagnosis and resolution.

Symptoms

Typical error output:

bash
Error: Exceeded memory limit for $lookup
$lookup stage exceeds 100MB limit
Consider using $lookup with pipeline and pipeline stages

Common Causes

  1. 1.Joining large collections without optimization
  2. 2.Unbounded $lookup returning too many matches
  3. 3.Memory limit for aggregation exceeded
  4. 4.Missing pipeline stages to limit joined data

Step-by-Step Fix

Step 1: Check Current State

javascript
db.collection.explain("executionStats").aggregate([...])
db.collection.stats()
db.adminCommand({getCmdLineOpts: 1})

Step 2: Identify Root Cause

javascript
rs.status()
db.serverStatus()
db.currentOp()

Step 3: Apply Primary Fix

javascript
// Use $lookup with pipeline to limit results
{
  $lookup: {
    from: "largeCollection",
    let: { foreignId: "$_id" },
    pipeline: [
      { $match: { $expr: { $eq: ["$_id", "$$foreignId"] } } },
      { $limit: 100 }
    ],
    as: "joined"
  }
}

Step 4: Apply Alternative Fix

```javascript // Alternative fix: Check configuration rs.conf() db.adminCommand({getCmdLineOpts: 1})

// Update settings db.adminCommand({setParameter: 1, parameter: value})

// Verify the fix db.serverStatus().metrics ```

Step 5: Verify the Fix

javascript
db.collection.aggregate([...]).pretty()
// Should complete without memory error

Common Pitfalls

  • Not checking replica set status before operations
  • Using unbounded queries on large collections
  • Forgetting to create indexes for query patterns
  • Ignoring memory limits in aggregation pipelines

Best Practices

  • Monitor replication lag and oplog size
  • Create indexes to support common queries
  • Use covered queries to reduce document scans
  • Implement retry logic for transient errors
  • MongoDB Replica Set Election
  • MongoDB Sharding Chunk Migration
  • MongoDB Aggregation Pipeline Error
  • MongoDB Connection Failed