Introduction

Java OutOfMemoryError: Direct buffer memory occurs when the JVM exhausts the memory allocated for NIO direct buffers, which are allocated outside the Java heap in native memory. Direct buffers provide efficient I/O operations by allowing direct transfer between buffer and native code (useful for NIO channels, Netty, database drivers) but are limited by -XX:MaxDirectMemorySize (defaults to -Xmx value). Unlike heap objects, direct buffers are not garbage collected immediately - they rely on Cleaner/PhantomReference for deallocation, which can lead to memory exhaustion if allocation rate exceeds cleanup rate. Common causes include allocating direct buffers without proper release, buffer pools not properly managed, memory leaks in native libraries, Netty or framework misconfiguration, high concurrency creating many direct buffers, direct buffer size exceeding MaxDirectMemorySize, and GC not triggering often enough to clean phantom references. The fix requires understanding direct buffer lifecycle, proper pool configuration, native memory tracking, and JVM tuning for direct buffer management. This guide provides production-proven techniques for debugging and fixing direct buffer memory errors across Java applications using NIO, Netty, and other native memory-intensive frameworks.

Symptoms

  • java.lang.OutOfMemoryError: Direct buffer memory in logs
  • java.lang.OutOfMemoryError: Cannot reserve X bytes for direct buffer
  • Application memory (RSS) exceeds heap size significantly
  • Direct buffer memory pool exhausted metrics
  • Netty: Too many outstanding allocations errors
  • Native memory grows continuously while heap stable
  • GC runs frequently but doesn't reclaim direct memory
  • Error occurs under high I/O load or concurrency
  • Direct buffer count much higher than expected
  • sun.misc.Unsafe allocation failures

Common Causes

  • Direct ByteBuffer allocated but not released
  • ByteBuffer pool exhausted (all buffers in use)
  • Memory leak in native library holding direct buffer references
  • Netty arena cache not properly configured
  • High allocation rate exceeds Cleaner throughput
  • Direct buffer size larger than MaxDirectMemorySize
  • Many concurrent connections each allocating buffers
  • FileChannel or SocketChannel not properly closed
  • Direct buffer references held after use (preventing GC)
  • JNI code allocating native memory without cleanup

Step-by-Step Fix

### 1. Diagnose direct buffer usage

Check current direct buffer memory:

```java // JMX bean for direct buffer statistics import java.lang.management.BufferPoolMXBean; import java.lang.management.ManagementFactory;

List<BufferPoolMXBean> pools = ManagementFactory.getPlatformMXBeans( BufferPoolMXBean.class);

for (BufferPoolMXBean pool : pools) { if ("direct".equals(pool.getName())) { System.out.println("Direct Buffer Pool:"); System.out.println(" Count: " + pool.getCount()); System.out.println(" Used: " + pool.getMemoryUsed() / 1024 / 1024 + " MB"); System.out.println(" Capacity: " + pool.getTotalCapacity() / 1024 / 1024 + " MB"); } }

// Or using reflection for HotSpot-specific access import sun.misc.VM; long directMemoryUsed = VM.maxDirectMemory() - VM.directMemoryUsed(); ```

Enable native memory tracking:

```bash # Enable NMT (Native Memory Tracking) java -XX:NativeMemoryTracking=summary -jar app.jar

# Or detailed tracking (more overhead) java -XX:NativeMemoryTracking=detail -jar app.jar

# Check native memory usage jcmd <pid> VM.native_memory summary

# Output shows: # Native Memory Tracking: # Total: reserved=2GB, committed=1.5GB # - Java Heap: reserved=1GB, committed=800MB # - Class: reserved=250MB, committed=100MB # - Thread: reserved=50MB, committed=50MB # - Code: reserved=250MB, committed=200MB # - GC: reserved=100MB, committed=80MB # - Internal: reserved=50MB, committed=40MB # - Direct: reserved=500MB, committed=400MB <-- Check this # - Other: reserved=100MB, committed=80MB

# Track changes over time jcmd <pid> VM.native_memory detail > nmt-dump-1.txt # ... run workload ... jcmd <pid> VM.native_memory detail > nmt-dump-2.txt # Compare the two dumps ```

Enable GC logging for direct buffer debugging:

```bash # GC logs show direct buffer cleanup java -Xlog:gc*:file=gc.log:time,level,tags -jar app.jar

# Java 8 java -XX:+PrintGCDetails -XX:+PrintGCDateStamps \ -Xloggc:gc.log -jar app.jar

# Look for: # - GC pauses (direct buffer cleanup happens during GC) # - Old Gen collection frequency # - Reference processing (Cleaner runs during GC) ```

### 2. Configure MaxDirectMemorySize

Increase direct memory limit:

```bash # Default: equals -Xmx (heap max) # If you need more direct memory:

java -Xmx4g \ -XX:MaxDirectMemorySize=2g \ -jar app.jar

# For Netty applications (common pattern) java -Xmx4g \ -XX:MaxDirectMemorySize=4g \ -Dio.netty.allocator.type=pooled \ -Dio.netty.maxDirectMemory=4g \ -jar app.jar

# Rule of thumb: # MaxDirectMemorySize = 0.5x to 1x heap size for typical apps # MaxDirectMemorySize = 1x to 2x heap size for I/O intensive apps ```

Monitor direct memory usage:

```java // Add monitoring endpoint @RestController public class DirectBufferMetrics {

@GetMapping("/metrics/direct-buffer") public Map<String, Object> getDirectBufferMetrics() { Map<String, Object> metrics = new HashMap<>();

List<BufferPoolMXBean> pools = ManagementFactory .getPlatformMXBeans(BufferPoolMXBean.class);

for (BufferPoolMXBean pool : pools) { if ("direct".equals(pool.getName())) { metrics.put("count", pool.getCount()); metrics.put("used_bytes", pool.getMemoryUsed()); metrics.put("used_mb", pool.getMemoryUsed() / 1024 / 1024); metrics.put("capacity_bytes", pool.getTotalCapacity()); } }

// Also report MaxDirectMemorySize try { Field f = sun.misc.VM.class.getDeclaredField("directMemory"); f.setAccessible(true); metrics.put("max_direct_memory", ((Number) f.get(null)).longValue()); } catch (Exception e) { metrics.put("max_direct_memory", "unknown"); }

return metrics; } } ```

### 3. Fix direct buffer leaks

Proper buffer release pattern:

```java // WRONG: Direct buffer without explicit cleanup public void processData() throws IOException { ByteBuffer buffer = ByteBuffer.allocateDirect(1024 * 1024); // 1MB channel.read(buffer); process(buffer); // Buffer not explicitly cleaned - relies on GC // If allocation rate > GC rate, OOM occurs! }

// CORRECT: Use try-with-resources for Closeable buffers public void processData() throws IOException { DirectByteBuffer buffer = null; try { buffer = new DirectByteBuffer(1024 * 1024); channel.read(buffer); process(buffer); } finally { if (buffer != null) { buffer.clean(); // Explicit cleanup } } }

// CORRECT: Use pool for repeated allocations private static final DirectBufferPool pool = new DirectBufferPool(100, 1024 * 1024);

public void processData() throws IOException { ByteBuffer buffer = pool.acquire(); try { channel.read(buffer); process(buffer); } finally { buffer.clear(); pool.release(buffer); // Return to pool } } ```

Netty buffer management:

```java // Netty ByteBuf with proper release // WRONG: ByteBuf not released public void handleRequest(ChannelHandlerContext ctx, Object msg) { ByteBuf buf = ctx.alloc().directBuffer(1024); buf.writeBytes(data); ctx.writeAndFlush(buf); // If write fails or is delayed, buffer may leak! }

// CORRECT: Use ReferenceCountUtil public void handleRequest(ChannelHandlerContext ctx, Object msg) { ByteBuf buf = ctx.alloc().directBuffer(1024); try { buf.writeBytes(data); ctx.writeAndFlush(buf.retain()); // Retain before async operation } finally { buf.release(); // Release our reference } }

// CORRECT: Use try-finally for complex operations ByteBuf buf = ctx.alloc().directBuffer(size); boolean success = false; try { // ... operations ... success = true; } finally { if (!success) { buf.release(); } } ```

Netty leak detection:

```bash # Enable Netty leak detection java -Dio.netty.leakDetection.level=PARANOID \ -Dio.netty.leakDetection.targetRecords=32 \ -jar app.jar

# Leak levels: DISABLED, SIMPLE, ADVANCED, PARANOID # PARANOID: Every allocation tracked (high overhead) # ADVANCED: Sample-based (recommended for production)

# Netty will log leak reports: # LEAK: ByteBuf.release() not called # Recent access records: # Created at: io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(...) # Last access: ... ```

### 4. Fix FileChannel/SocketChannel issues

Proper channel cleanup:

```java // WRONG: Channel not closed public void readFile(String path) throws IOException { FileChannel channel = FileChannel.open(Paths.get(path)); ByteBuffer buffer = ByteBuffer.allocateDirect(4096); channel.read(buffer); // Channel never closed - direct buffer held! }

// CORRECT: Try-with-resources public void readFile(String path) throws IOException { try (FileChannel channel = FileChannel.open(Paths.get(path))) { ByteBuffer buffer = ByteBuffer.allocateDirect(4096); channel.read(buffer); process(buffer); } // Channel and associated buffers cleaned up }

// CORRECT: Explicit close with cleanup FileChannel channel = null; try { channel = FileChannel.open(Paths.get(path)); ByteBuffer buffer = ByteBuffer.allocateDirect(4096); channel.read(buffer); process(buffer); } finally { if (channel != null) { channel.close(); // Releases associated direct buffers } } ```

### 5. Tune direct buffer cleanup

Force direct buffer cleanup:

```java // Cleaner-based cleanup (Java 9+) import java.lang.ref.Cleaner;

public class DirectBufferResource implements AutoCloseable { private static final Cleaner cleaner = Cleaner.create(); private final ByteBuffer buffer; private final Cleaner.Cleanable cleanable;

public DirectBufferResource(int size) { this.buffer = ByteBuffer.allocateDirect(size); this.cleanable = cleanable.register(this, () -> { // Cleanup action when DirectBufferResource is GC'd cleanDirectBuffer(buffer); }); }

private static void cleanDirectBuffer(ByteBuffer buffer) { if (buffer instanceof sun.nio.ch.DirectBuffer) { sun.nio.ch.DirectBuffer db = (sun.nio.ch.DirectBuffer) buffer; if (db.cleaner() != null) { db.cleaner().clean(); } } }

@Override public void close() { cleanable.clean(); // Explicit cleanup } }

// Usage with try-with-resources try (DirectBufferResource resource = new DirectBufferResource(1024 * 1024)) { use(resource.getBuffer()); } // Automatically cleaned up ```

Force GC for cleanup (emergency only):

```java // Emergency: Force GC to trigger Cleaner // Only use as last resort - very expensive! public static void forceDirectBufferCleanup() { System.gc(); // Hint to JVM to run GC try { Thread.sleep(100); // Give Cleaner time to run } catch (InterruptedException e) { Thread.currentThread().interrupt(); } }

// Better: Monitor and alert before OOM @Scheduled(every = "1m") public void monitorDirectMemory() { long used = getDirectMemoryUsed(); long max = getMaxDirectMemory(); double ratio = (double) used / max;

if (ratio > 0.8) { log.warn("Direct memory usage high: {}%", ratio * 100); // Trigger alert, consider scaling } } ```

### 6. Configure buffer pools

Netty pool configuration:

```java // Netty PooledByteBufAllocator configuration import io.netty.buffer.PooledByteBufAllocator; import io.netty.buffer.PooledByteBufAllocatorMetric;

// Configure allocator PooledByteBufAllocator allocator = new PooledByteBufAllocator( true, // preferDirect 10, // nHeapArena - number of heap arenas 10, // nDirectArena - number of direct arenas 8192, // pageSize - size of memory page 11, // maxOrder - max buffer order (2^11 = 2MB max) 0, // smallCacheSize 0, // normalCacheSize true, // useCacheForAllThreads 0 // directMemoryCacheAlignment );

// Monitor pool usage PooledByteBufAllocatorMetric metric = allocator.metric(); System.out.println("Direct arenas: " + metric.numDirectArenas()); System.out.println("Thread local caches: " + metric.numThreadLocalCaches());

// Set via system properties // -Dio.netty.allocator.type=pooled // -Dio.netty.allocator.numDirectArenas=10 // -Dio.netty.allocator.pageSize=8192 ```

Custom buffer pool:

```java // Custom direct buffer pool public class DirectBufferPool { private final ArrayBlockingQueue<ByteBuffer> pool; private final int bufferSize;

public DirectBufferPool(int poolSize, int bufferSize) { this.bufferSize = bufferSize; this.pool = new ArrayBlockingQueue<>(poolSize);

// Pre-allocate buffers for (int i = 0; i < poolSize; i++) { pool.offer(ByteBuffer.allocateDirect(bufferSize)); } }

public ByteBuffer acquire() throws InterruptedException { ByteBuffer buffer = pool.poll(1, TimeUnit.SECONDS); if (buffer == null) { // Pool exhausted - allocate temporarily or reject throw new IllegalStateException("Buffer pool exhausted"); } buffer.clear(); return buffer; }

public void release(ByteBuffer buffer) { buffer.clear(); if (!pool.offer(buffer)) { // Pool full - buffer will be GC'd } }

public int getAvailableBuffers() { return pool.size(); } } ```

Prevention

  • Always close FileChannel/SocketChannel with try-with-resources
  • Use buffer pools for repeated allocations
  • Enable Netty leak detection in staging environment
  • Monitor direct buffer memory with alerts
  • Set appropriate MaxDirectMemorySize based on workload
  • Use pooled allocators (Netty PooledByteBufAllocator)
  • Implement backpressure when buffer pool exhausted
  • Regular NMT analysis to track native memory trends
  • Document direct buffer usage in code comments
  • Load test with production-like I/O patterns
  • **Java OutOfMemoryError heap space**: Heap memory exhaustion
  • **Java OutOfMemoryError Metaspace**: Class metadata exhaustion
  • **Java StackOverflowError**: Infinite recursion
  • **Java GC overhead limit exceeded**: Too much time in GC