Introduction

Linux core dumps capture a process's memory state at the moment of a crash, providing essential debugging information. The kernel.core_pattern sysctl controls where core dumps are written. When it is configured to pipe to an external handler (like |/usr/lib/systemd/systemd-coredump), core dumps may not appear in the expected location, or may not be generated at all if the pipe handler fails, the disk is full, or the storage limit is reached.

Symptoms

  • Application crashes but no core file appears in the working directory
  • ulimit -c unlimited is set but no core dump is produced
  • sysctl kernel.core_pattern shows a pipe: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
  • coredumpctl list shows no entries despite known crashes
  • dmesg shows core dump handler returned 1 or failed to create core file

Common Causes

  • core_pattern pipes to systemd-coredump but the journal storage is limited
  • Disk partition for core dumps is full
  • Storage=none in coredump.conf disabling core dump collection
  • ProcessSizeMax limit in coredump.conf too small for the crashing process
  • Application runs in a container where core_pattern pipe target does not exist

Step-by-Step Fix

  1. 1.Check current core pattern:
  2. 2.```bash
  3. 3.sysctl kernel.core_pattern
  4. 4.# Pipe to external program: |/path/to/handler
  5. 5.# Direct file write: /path/to/core.%e.%p
  6. 6.`
  7. 7.Check systemd-coredump configuration:
  8. 8.```bash
  9. 9.cat /etc/systemd/coredump.conf
  10. 10.# Check for Storage= and ProcessSizeMax= settings
  11. 11.coredumpctl list
  12. 12.coredumpctl info <pid>
  13. 13.`
  14. 14.Configure systemd-coredump to store cores:
  15. 15.```bash
  16. 16.sudo nano /etc/systemd/coredump.conf
  17. 17.`
  18. 18.Add or modify:
  19. 19.```ini
  20. 20.[Coredump]
  21. 21.Storage=external
  22. 22.Compress=yes
  23. 23.ProcessSizeMax=2G
  24. 24.ExternalSizeMax=2G
  25. 25.`
  26. 26.Then reload:
  27. 27.```bash
  28. 28.sudo systemctl restart systemd-coredump.socket
  29. 29.`
  30. 30.Alternatively, configure direct core file writing:
  31. 31.```bash
  32. 32.sudo mkdir -p /var/cores
  33. 33.sudo chmod 777 /var/cores
  34. 34.echo "/var/cores/core.%e.%p.%t" | sudo tee /proc/sys/kernel/core_pattern
  35. 35.echo "kernel.core_pattern=/var/cores/core.%e.%p.%t" | sudo tee -a /etc/sysctl.conf
  36. 36.`
  37. 37.Ensure ulimit allows core dumps:
  38. 38.```bash
  39. 39.ulimit -c
  40. 40.# If 0, enable unlimited:
  41. 41.ulimit -c unlimited
  42. 42.# For systemd services, add to unit file:
  43. 43.# [Service]
  44. 44.# LimitCORE=infinity
  45. 45.`
  46. 46.Extract a coredump from systemd-coredump for analysis:
  47. 47.```bash
  48. 48.# List available coredumps
  49. 49.coredumpctl list

# Get the latest coredump for a specific executable coredumpctl dump /usr/bin/myapp -o /tmp/myapp.core

# Analyze with gdb gdb /usr/bin/myapp /tmp/myapp.core (gdb) bt ```

Prevention

  • Configure Storage=external in coredump.conf to always save core dumps
  • Set adequate ExternalSizeMax and ProcessSizeMax limits (at least 1GB)
  • Monitor /var/lib/systemd/coredump/ directory size and implement rotation
  • Include core dump configuration validation in server provisioning scripts
  • For containers, mount the host's core_pattern pipe handler or configure direct file writing