Fix Pulsar Broker Out Of Memory

What's Actually Happening

Apache Pulsar broker runs out of heap memory and crashes or becomes unresponsive. The broker cannot handle its assigned topic bundles.

The Error You'll See

OOM in logs:

```bash $ journalctl -u pulsar-broker | grep -i "out of memory"

java.lang.OutOfMemoryError: Java heap space ```

Broker crash:

```bash $ systemctl status pulsar-broker

Active: failed (Result: exit-code) Main PID: 12345 (code=exited, status=137) ```

Memory warning:

WARN  io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before GC

Bookie errors:

ERROR org.apache.bookkeeper.bookie.Bookie - Out of memory in bookie

Why This Happens

1. 1.JVM heap too small - Insufficient memory for workload
2. 2.Too many topics - Broker handling too many topic bundles
3. 3.Large messages - Messages too big for buffer
4. 4.Memory leak - Application not releasing memory
5. 5.Backlog large - Large message backlog in memory
6. 6.Offloading not configured - No tiered storage

Step 1: Check Broker Memory

```bash # Check broker process memory: ps aux | grep pulsar-broker

# Check JVM heap settings: ps aux | grep pulsar | grep Xmx

# Check current memory usage: jstat -gc <pid> jstat -gcutil <pid>

# Get detailed heap info: jmap -heap <pid>

# Check for memory leaks: jcmd <pid> GC.heap_info

# Check broker stats: curl http://localhost:8080/admin/v2/brokers/stats

# Memory stats from admin: curl http://localhost:8080/metrics | grep -i memory ```

Step 2: Increase JVM Heap

```bash # Check current heap in pulsar_env.sh: cat /etc/pulsar/pulsar_env.sh | grep PULSAR_MEM

# Default: # PULSAR_MEM="-Xms2g -Xmx2g -XX:MaxDirectMemorySize=2g"

# Increase heap: export PULSAR_MEM="-Xms4g -Xmx4g -XX:MaxDirectMemorySize=4g"

# Or edit pulsar_env.sh: PULSAR_MEM="-Xms4g -Xmx4g -XX:MaxDirectMemorySize=4g"

# For larger deployments: PULSAR_MEM="-Xms8g -Xmx8g -XX:MaxDirectMemorySize=4g"

# Restart broker: systemctl restart pulsar-broker

# Verify new settings: ps aux | grep pulsar | grep Xmx ```

Step 3: Check Topic Load

```bash # List topics on broker: curl http://localhost:8080/admin/v2/brokers/localhost:8080/ownedNamespaces

# Check topic bundle assignment: curl http://localhost:8080/admin/v2/bundles/public/default

# List all topics: curl http://localhost:8080/admin/v2/persistent/public/default

# Check topic stats: curl http://localhost:8080/admin/v2/persistent/public/default/mytopic/stats

# Topics with large backlog: curl http://localhost:8080/admin/v2/persistent/public/default/mytopic/stats | jq '.msgBacklog'

# Unload topics from overloaded broker: curl -X PUT http://localhost:8080/admin/v2/namespaces/public/default/unload

# Or unload specific bundle: curl -X PUT http://localhost:8080/admin/v2/namespaces/public/default/bundle/0x00000000_0xffffffff/unload ```

Step 4: Configure Topic Offloading

```bash # Enable offloading in broker.conf: cat >> /etc/pulsar/broker.conf << EOF # Offloading configuration managedLedgerOffloadDriver=S3 managedLedgerMinLedgerRolloverTimeMinutes=10 managedLedgerMaxEntriesPerLedger=50000

# Offload when backlog > 10GB managedLedgerOffloadThresholdInBytes=10737418240

# S3 configuration s3ManagedLedgerOffloadBucket=your-bucket s3ManagedLedgerOffloadServiceEndpoint=s3.amazonaws.com s3ManagedLedgerOffloadRegion=us-east-1 EOF

# Set S3 credentials: export AWS_ACCESS_KEY_ID=your-key export AWS_SECRET_ACCESS_KEY=your-secret

# Or in broker.conf: s3ManagedLedgerOffloadCredentialId=your-key s3ManagedLedgerOffloadCredentialSecret=your-secret

# Restart broker: systemctl restart pulsar-broker

# Manually offload topic: curl -X PUT http://localhost:8080/admin/v2/persistent/public/default/mytopic/offload ```

Step 5: Limit Topics Per Broker

```bash # Check current limits: curl http://localhost:8080/admin/v2/brokers/localhost:8080/configuration

# Set namespace bundle limits: # In broker.conf: numHttpServerThreads=8 numIOThreads=8

# Limit bundles per broker: # defaultNumberOfNamespaceBundles=4

# Reduce load per broker: curl -X PUT http://localhost:8080/admin/v2/namespaces/public/default/bundles -d '{"numBundles": 8}'

# Split bundles to distribute load: curl -X PUT http://localhost:8080/admin/v2/namespaces/public/default/split

# Check bundle distribution: curl http://localhost:8080/admin/v2/bundles/public/default ```

Step 6: Optimize Message Size

```bash # Check max message size: curl http://localhost:8080/admin/v2/brokers/localhost:8080/configuration | jq '.maxMessageSize'

# Default: 5MB

# Large messages consume more memory: # If sending large messages, chunk them

# In producer: Producer<byte[]> producer = client.newProducer() .topic("mytopic") .enableChunking(true) .maxMessageSize(5 * 1024 * 1024) // 5MB chunks .create();

# Or use compression: Producer<byte[]> producer = client.newProducer() .topic("mytopic") .compressionType(CompressionType.LZ4) .create();

# Check message sizes: curl http://localhost:8080/admin/v2/persistent/public/default/mytopic/stats | jq '.averageMsgSize' ```

Step 7: Configure Cache Settings

```bash # Pulsar caches messages in memory for readers

# In broker.conf: # Message cache managedLedgerCacheSizeMB=1024 managedLedgerCacheEvictionWatermark=0.9

# Reduce cache if memory tight: managedLedgerCacheSizeMB=512

# Entry cache per ledger: managedLedgerCursorMaxEntriesPerLedger=50000

# In bookkeeper.conf: journalMaxGroupWaitMSec=1 journalSyncData=true numAddWorkerThreads=4 numReadWorkerThreads=4 numJournalCallbackThreads=4

# Restart after changes: systemctl restart pulsar-broker systemctl restart pulsar-bookie ```

Step 8: Check for Memory Leaks

```bash # Enable GC logging: export PULSAR_GC_OPTS="-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/pulsar/gc.log"

# Restart broker: systemctl restart pulsar-broker

# Monitor GC logs: tail -f /var/log/pulsar/gc.log

# Look for frequent full GC: grep "Full GC" /var/log/pulsar/gc.log

# Take heap dump: jcmd <pid> GC.heap_dump /tmp/pulsar-heap.hprof

# Analyze with VisualVM or MAT: # Check for large objects, leaks

# Check direct buffer usage: jcmd <pid> VM.native_memory summary

# Netty leak detection: export PULSAR_OPTS="-Dio.netty.leakDetection.level=advanced" ```

Step 9: Monitor Broker Metrics

```bash # Get broker metrics: curl http://localhost:8080/metrics

# Key memory metrics: # pulsar_broker_memory_heap_used # pulsar_broker_memory_heap_max # pulsar_broker_backlog_quota_exceeded

# Create monitoring script: cat << 'EOF' > /usr/local/bin/monitor-pulsar.sh #!/bin/bash

echo "=== Broker Memory ===" curl -s http://localhost:8080/metrics | grep -i "heap|direct"

echo "" echo "=== Topic Count ===" curl -s http://localhost:8080/admin/v2/persistent/public/default | jq 'length'

echo "" echo "=== Owned Namespaces ===" curl -s http://localhost:8080/admin/v2/brokers/localhost:8080/ownedNamespaces | jq '.'

echo "" echo "=== Backlog Stats ===" curl -s http://localhost:8080/admin/v2/persistent/public/default | jq -r '.[]' | while read topic; do backlog=$(curl -s http://localhost:8080/admin/v2/persistent/$topic/stats | jq '.msgBacklog') if [ "$backlog" -gt 100000 ]; then echo "High backlog: $topic = $backlog" fi done EOF

chmod +x /usr/local/bin/monitor-pulsar.sh

# Prometheus alerts: - alert: PulsarBrokerOOM expr: pulsar_broker_memory_heap_used / pulsar_broker_memory_heap_max > 0.9 for: 5m labels: severity: critical annotations: summary: "Pulsar broker heap usage > 90%" ```

Step 10: Scale Brokers

```bash # If single broker cannot handle load, add more:

# On new broker: # Install Pulsar broker # Configure to connect to same ZooKeeper and BookKeeper

# Start new broker: systemctl start pulsar-broker

# Check broker registration: curl http://localhost:8080/admin/v2/brokers

# Trigger load balance: curl -X PUT http://localhost:8080/admin/v2/brokers/leader-broker

# Topics will automatically distribute across brokers

# Check load distribution: curl http://localhost:8080/admin/v2/brokers/leader-broker/brokerStatus

# For Kubernetes: kubectl scale statefulset pulsar-broker --replicas=3 ```

Pulsar Broker Memory Checklist

Verify the Fix

```bash # After increasing memory and optimizing

# 1. Check heap usage curl http://localhost:8080/metrics | grep heap_used // < 80% of max

# 2. Check broker stable systemctl status pulsar-broker // Active: running

# 3. Test message production pulsar-client produce mytopic -m "test message" // Published successfully

# 4. Monitor GC tail /var/log/pulsar/gc.log // No frequent full GC

# 5. Check offloading curl http://localhost:8080/admin/v2/persistent/public/default/mytopic/offload // Offload working

# 6. Verify no OOM journalctl -u pulsar-broker | grep -i "out of memory" // No OOM errors ```

Related Issues

• [Fix Pulsar Broker Unavailable](/articles/fix-pulsar-broker-unavailable)
• [Fix Pulsar Backlog Quota Exceeded](/articles/fix-pulsar-backlog-quota-exceeded)
• [Fix Kafka Broker Not Starting](/articles/fix-kafka-broker-not-starting)