Introduction

Linux allows mounting a filesystem on top of an existing mount point, which effectively hides the original content. This is called a mount overlay or shadowed mount. When a bind mount is accidentally placed over an existing mount, the underlying data becomes inaccessible but continues consuming disk space. This is notoriously difficult to diagnose because ls shows the overlay content while the hidden data is silently inaccessible.

Symptoms

  • Disk usage reported by df does not match du calculations
  • Expected files or directories appear empty or missing
  • Application fails with File not found despite data existing on disk
  • Database tables disappear after a restart (data directory mounted over)

Common Causes

  • Docker volume mount conflicts where container mount point matches host mount
  • fstab entry mounting over an existing filesystem without checking
  • Kubernetes emptyDir or hostPath volumes shadowing existing data directories
  • Manual mount --bind command overlaying an active mount point
  • Initramfs scripts mounting root filesystem incorrectly

Step-by-Step Fix

  1. 1.Detect hidden (shadowed) mount points:
  2. 2.```bash
  3. 3.# List all mounts and look for overlaps
  4. 4.findmnt --all --tree

# Check for mounts on the same target findmnt /data findmnt /data/subdir ```

  1. 1.Identify the hidden content using a bind mount trick:
  2. 2.```bash
  3. 3.# Create an alternate access point to the underlying filesystem
  4. 4.sudo mkdir /tmp/hidden-view
  5. 5.sudo mount --bind /dev/sda1 /tmp/hidden-view
  6. 6.ls /tmp/hidden-view/data/
  7. 7.`
  8. 8.View the actual mount stack:
  9. 9.```bash
  10. 10.cat /proc/self/mountinfo | grep "/data"
  11. 11.# Shows all mounts in the stack with their parent IDs
  12. 12.`
  13. 13.Unmount the overlay to reveal hidden data:
  14. 14.```bash
  15. 15.sudo umount /data/subdir
  16. 16.# If busy, find what is using it:
  17. 17.sudo lsof +D /data/subdir
  18. 18.sudo fuser -m /data/subdir
  19. 19.sudo umount -l /data/subdir
  20. 20.`
  21. 21.Reorganize fstab to prevent future conflicts:
  22. 22.```bash
  23. 23.sudo nano /etc/fstab
  24. 24.# Ensure mount points do not overlap
  25. 25.# Use distinct, non-nested paths
  26. 26.`
  27. 27.Remount in correct order:
  28. 28.```bash
  29. 29.sudo mount -a
  30. 30.findmnt --verify
  31. 31.`

Prevention

  • Use findmnt --verify after editing fstab to detect overlapping mounts
  • Implement mount point checks in deployment scripts: mountpoint -q /data || mkdir /data
  • Avoid nested mount points; use sibling directories instead
  • Use x-mount.mkdir=0755 in fstab to auto-create directories
  • Document mount hierarchy and review before adding new entries