Segfault: Race condition in concurrent cache operations with high thread counts

[TylerMills]

Describe the bug

MergerFS v2.40.2 experienced a segmentation fault during heavy I/O operations from SABnzbd (Usenet downloader). The segfault caused the MergerFS mount to become unresponsive with "Transport endpoint is not connected" errors, which then caused all Docker containers using the mount to fail.

Please be sure to use latest release of mergerfs to ensure the issue still exists. Not your distro's latest but the latest official release.

Using mergerfs v2.40.2 (latest official release as of March 1, 2024)

To Reproduce

Steps to reproduce the behavior:

1. MergerFS mount configured with 4 branches (1 SSD cache + 1 ZFS pool + 2 NFS shares)
2. Heavy concurrent write operations via SABnzbd downloading multiple large files simultaneously
3. Cache SSD was at 73% capacity with ~68GB of incomplete downloads
4. After approximately 30-60 minutes of heavy downloading, MergerFS segfaulted

MergerFS mount configuration from /etc/fstab:

/mnt/cluster-cache-1:/mnt/cluster-c:/mnt/cluster-a:/mnt/cluster-b /mnt/storage-cluster fuse.mergerfs allow_other,use_ino,cache.files=full,dropcacheonclose=true,category.create=ff,minfreespace=16G,uid=1000,gid=1000,cache.writeback=true,cache.statfs=60,cache.attr=60,cache.entry=60,cache.negative_entry=60,cache.symlinks=true,cache.readdir=true,parallel-direct-writes=true,read-thread-count=16,process-thread-count=16,readahead=1024,security_capability=false,posix_acl=false,async_read=false,_netdev,nonempty,x-systemd.automount,noauto,nofail,x-systemd.requires-mounts-for=/mnt/cluster-cache-1,x-systemd.requires-mounts-for=/mnt/cluster-a,x-systemd.requires-mounts-for=/mnt/cluster-b,x-systemd.requires-mounts-for=/mnt/cluster-c 0 0

Expected behavior

MergerFS should handle high concurrent I/O operations without crashing, even under heavy load from applications like SABnzbd.

System information:

• OS, kernel version:

Linux home-server 6.11.0-25-generic #25~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 15 17:20:50 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

• mergerfs version:

mergerfs v2.40.2

• mergerfs settings:

  • Branches: /mnt/cluster-cache-1:/mnt/cluster-c:/mnt/cluster-a:/mnt/cluster-b
  • Create policy: ff (fill first)
  • Min free space: 16G
  • Cache settings: cache.files=full, cache.writeback=true, various cache timeouts set to 60
  • Thread counts: read-thread-count=16, process-thread-count=16
  • Other: parallel-direct-writes=true, readahead=1024

• List of drives, filesystems, & sizes:

Filesystem                                Size  Used Avail Use% Mounted on
/dev/nvme0n1p3                            1.8T  1.3T  472G  73% /mnt/cluster-cache-1
cluster-c                                 130T  2.1T  128T   2% /mnt/cluster-c
67.2xx.xxx.xxx:/var/nfs/shared/cluster_a   88T  1.1T   87T   2% /mnt/cluster-a
67.2xx.xxx.xx:/var/nfs/shared/cluster_b   88T   59T   29T  68% /mnt/cluster-b

Filesystem types:

• /mnt/cluster-cache-1: ext4 on NVMe SSD
• /mnt/cluster-c: ZFS pool (10x HDD RAIDZ)
• /mnt/cluster-a & /mnt/cluster-b: NFS v3 shares from Ubiquiti UNAS Pro

Crash Information:

Kernel log showing segfault:

[Fri May 23 00:22:56 2025] fuse.process[726341]: segfault at 762726058079 ip 000063e52d992140 sp 00007620cb7eb408 error 4 in mergerfs[73140,63e52d92d000+7c000] likely on CPU 25 (core 41, socket 0)
[Fri May 23 02:12:25 2025] fuse.process[791721]: segfault at 79904d7c2c61 ip 00005f2109fef140 sp 00007998633ec408 error 4 in mergerfs[73140,5f2109f8a000+7c000] likely on CPU 25 (core 41, socket 0)

Signal from crash dump: 11 (SIGSEGV)

Command line at crash:

mergerfs /mnt/cluster-cache-1:/mnt/cluster-c:/mnt/cluster-a:/mnt/cluster-b /mnt/storage-cluster -o rw,allow_other,use_ino,cache.files=full,dropcacheonclose=true,category.create=ff,minfreespace=64G,uid=1000,gid=1000,cache.writeback=true,cache.statfs=60,cache.attr=60,cache.entry=60,cache.negative_entry=60,cache.symlinks=true,cache.readdir=true,parallel-direct-writes=true,read-thread-count=16,process-thread-count=16,readahead=1024,security_capability=false,posix_acl=false,async_read=false,nonempty,dev,suid

Additional context

• The crash occurred during heavy write operations from SABnzbd downloading multiple large files
• A cache mover container (ghcr.io/monstermuffin/mergerfs-cache-mover) was configured to run every 30 minutes but had exited earlier
• The system had been running for several days without issues before this crash
• After the crash, attempts to remount MergerFS failed with "Transport endpoint is not connected" errors
• A full crash dump is available at /var/crash/_usr_bin_mergerfs.0.crash (101MB)
• The crash appears to consistently happen on CPU 25 (core 41, socket 0)

The timing suggests the crash may be related to high concurrent I/O operations, possibly a race condition when handling multiple write operations across different backend filesystems (local SSD, ZFS, and NFS).

Crash Dump Analysis

Analysis of the core dump reveals:

1. Memory Corruption Evidence:

   • Register values show corrupted pointers: rax=0xaac1296ebc486919, rbx/rcx/rdi=0xee091b86ee1d6979
   • These values appear to be garbage data indicating memory corruption
   • The instruction pointer (rip=0x63e52d992140) was at a valid code location in mergerfs

2. Thread State at Crash (34 threads total):

   • Multiple threads performing posix_fadvise64 with POSIX_FADV_DONTNEED (advise=4) on various file descriptors
   • Several threads reading from fd=4 (likely the FUSE device) with 1MB buffers
   • One thread performing fdatasync on fd=135
   • One thread performing pwrite64
   • Process was in "D" state (disk sleep) with 34 threads, using ~89MB RSS

3. Concurrent I/O Pattern:

   • Many threads simultaneously calling posix_fadvise to drop cache pages
   • This suggests aggressive cache management under memory pressure
   • The combination of high thread counts and cache operations likely triggered a race condition

4. GDB Stack Traces (abbreviated):

   Thread 1 (crashed): #0  0x000063e52d992140 in ?? ()
   Thread 25: #0  __GI_fdatasync (fd=135)
   Thread 24: #0  __libc_pwrite64 (fd=72, count=3593)
   Threads 23-14,12-5: #0  __GI___libc_read (fd=4, nbytes=1052672)
   Threads 34-26,13: #0  __GI___posix_fadvise64_l64 (advise=4)
   

Likely Root Cause:
The crash appears to be a race condition in the cache management code when multiple threads simultaneously attempt to drop cache pages (posix_fadvise with DONTNEED) under heavy I/O load. The combination of:

• High thread counts (16 read + 16 process threads)
• Aggressive caching (cache.files=full, cache.writeback=true)
• parallel-direct-writes=true
• dropcacheonclose=true

Creates conditions where memory structures can be corrupted during concurrent cache operations.

[trapexit]

Thank you for the detailed report. I'm not sure about your analysis in so far as "cache management" here is called when an open file is released. mergerfs has no part in the actual caching. It's all normal syscalls and the mergerfs data structures used here are allocated per open'ed file request and release is only called when it is done with the file. There should be no possibility of race conditions with the data managed in that flow. I think it would be more likely that it is some shared/reused resource. Curiously this is the only segfault to be reported in a long time and a number of users have fairly threaded and aggressive setups. I'm not saying isn't mergerfs but given I've seen mergerfs crash on users with bad RAM and bad CPUs I'm not going to rule it out as a possibility till I can reproduce.

Wondering about your setup. Have you found that using cache.files=full and enabling writeback cache help with performance? Pretty much 100% of the time, on systems of all sizes, I find direct_io (aka cache.files=off) to be overall more performant and writeback of minimal use if the app is well written. In fact it's a known issue in the FUSE community that page caching is slower and something we/they want to investigate further.

BTW... parallel-direct-writes has no impact if direct io is not enabled (cache.files=off). Its enabled by default but doesn't mean anything here.

I'll try to setup up an instance with your config and see if I can break it with a synthetic load.

[HomebrewDotNET]

I'm currently also experiencing this on v2.41.1. Similar setup to OP. Using mergerfs to pool nvme's/sata's for cache/working directories. Essentially high io is expected all the time (Also includes torrenting, logging, ...)
The mergerfs pools are exposed to vm's using viriofs on Proxmox.

The last weeks I'm constantly getting ls: cannot access '/srv/storehouse': Transport endpoint is not connected type of errors
Since they are exposed via virtiofs I can't just do a simple remount and have to reboot all vm's
Noticed this in dmesg:
fuse.read[6918]: segfault at 28 ip 000056add2227261 sp 00007896360cdbc0 error 4 in mergerfs[54261,56add21e8000+109000] likely on CPU 11 (core 13, socket 0) [49828.302248] Code: 56 41 55 41 54 55 53 48 89 f3 48 81 ec 98 00 00 00 64 48 8b 2c 25 28 00 00 00 48 89 ac 24 88 00 00 00 48 89 d5 48 85 f6 74 47 <8b> 7b 28 48 89 ee e8 44 18 fc ff 83 f8 ff 75 09 e8 fa 0d fc ff 8b

Fstab for above mount looks like this:
/mnt/hot/lvm/lv-front-store:/mnt/hot/nvme/samsung_970_pro:/mnt/hot/nvme/kingston_fury_renegade:/mnt/warm/nvme/lexar_NM790:/mnt/warm/nvme/samsung_970_evo_plus_one:/mnt/warm/nvme/samsung_970_evo_plus_two:/mnt/warm/nvme/samsung_970_evo:/mnt/cold/nvme/kingston_kc3000:/mnt/cold/ssd/Micron_5200Eco:/mnt/cold/ssd/WDRed_SA500_one:/mnt/cold/ssd/WDRed_SA500_two:/mnt/cold/ssd/WDRed_SA500_three /srv/storehouse mergerfs defaults,nofail,readahead=4096,fuse-msg-size=4096,allow_other,moveonenospc=true,inodecalc=path-hash,posix-acl=true,func.mkdir=all,func.readdir=cor:12,category.create=ff,category.action=epall,category.search=newest,noauto,x-systemd.wants-mounts-for==/mnt/hot/nvme/samsung_970_pro,x-systemd.wants-mounts-for==/mnt/hot/nvme/kingston_fury_renegade,x-systemd.wants-mounts-for==/mnt/hot/lvm/lv-front-store,x-systemd.wants-mounts-for==/mnt/warm/nvme/lexar_NM790,x-systemd.wants-mounts-for==/mnt/warm/nvme/samsung_970_evo_plus_one,x-systemd.wants-mounts-for==/mnt/warm/nvme/samsung_970_evo_plus_two,x-systemd.wants-mounts-for==/mnt/warm/nvme/samsung_970_evo,x-systemd.wants-mounts-for==/mnt/cold/nvme/kingston_kc3000,x-systemd.wants-mounts-for==/mnt/cold/ssd/Micron_5200Eco,x-systemd.wants-mounts-for==/mnt/cold/ssd/WDRed_SA500_one,x-systemd.wants-mounts-for==/mnt/cold/ssd/WDRed_SA500_two,x-systemd.wants-mounts-for==/mnt/cold/ssd/WDRed_SA500_three,x-systemd.automount,minfreespace=100G,passthrough.io=rw,read-thread-count=-4,dropcacheonclose=false,func.getattr=newest 0 0

Not really sure if there's maybe a workaround already?

[trapexit]

Unless someone provides me with some way to reproduce the issue there isn't a whole lot I can do or justify the time investigating. v2.40.2 was out for more than a year used in high IO usecases without similar incidents being reported so the issue is unique to these setups or maybe even there are hardware issues.

[HomebrewDotNET]

Well if there's anything I can provide please let me know. Maybe it's an interaction with exposing mergerfs using virtiofs? They are both fuse based iirc so not sure how that would interact. All io also goes through virtiofs. Before the last crash a few minutes ago I saw IO wait spiking for kworker processes and mergerfs. Strange thing is I've been running on rc2 and rc3 when they came out without issues. Only thing I really added was passthrough.io and enable cache auto for virtiofs. Gonna try and run without that to see if it changes anything

[HomebrewDotNET]

Might aswell post the parameters I'm using for virtiofs: /usr/libexec/virtiofsd-bin --allow-mmap --thread-pool-size=16 --writeback --inode-file-handles=prefer --fd=17 --shared-dir=/srv/admin --xattr --announce-submounts --allow-direct-io --inode-file-handles=prefer --syslog
Thing here is direct-io is also enabled

[trapexit]

Well if there's anything I can provide please let me know.

As I point out in the support docs what I need is a minimal reproduction.

[damientkyt]

I just faced this issue (Transport endpoint is not connected) after mergerfs running fine for months.

[1499109.470781] fuse.read[3890646]: segfault at 28 ip 00005c7bfbe82a80 sp 0000747b1f9ecb20 error 4 in mergerfs[46a80,5c7bfbe4f000+c8000] likely on CPU 15 (core 7, socket 0)
[1508166.651510] fuse.read[802074]: segfault at 28 ip 000061598bb34a80 sp 000073bb04812b20 error 4 in mergerfs[46a80,61598bb01000+c8000] likely on CPU 5 (core 5, socket 0)
[1539530.866497] fuse.read[972127]: segfault at 28 ip 00005e924917b314 sp 0000734e3d7f9bd0 error 4 in mergerfs[48314,5e9249147000+ea000] likely on CPU 5 (core 5, socket 0)

Things I did today:

1. SSH into mergerfs pool to upload 5TB worth of mixed media files
2. apt update
3. ran filebrowser quantum installed using pve community scripts to view files

This error happened 3 times. Each time I just had to re-mount the mergerfs pool and restart the LXC/VMs for it to work again.

[trapexit]

@damientkyt A report without sufficient details is not useful.