Repository navigation
duperemove 0.15.2 hangs #407
Description
duperemove 0.15.2 (installed from nixpkgs) was running for a while but then hung for a week at 100% CPU usage.
duperemove -rd --cpu-threads=1 --io-threads=1 --dedupe-options=partial --hashfile=/root/duperemove.db /mnt /var/lib/mydir
The last log line was:
Dedupe 1 extents (id: bc6fd1db) with target: (0, 6413089905), "/var/lib/mydir/somelargefile"
I tried attaching an strace and it just printed half of a FUTEX_WAIT and never anything more.
I killed it and pulled the stack traces from the core dump:
Thread 3 (Thread 0x710b27fff6c0 (LWP 607050)):
#0 0x0000710b2852320d in syscall () from /nix/store/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61/lib/libc.so.6
#1 0x0000710b28ae2457 in g_cond_wait () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#2 0x0000710b28a720ac in g_async_queue_pop_intern_unlocked () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#3 0x0000710b28ae2ee4 in g_thread_pool_spawn_thread () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#4 0x0000710b28ae2c2e in g_thread_proxy () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#5 0x0000710b2849dd53 in start_thread () from /nix/store/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61/lib/libc.so.6
#6 0x0000710b2852563c in __clone3 () from /nix/store/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61/lib/libc.so.6
Thread 2 (Thread 0x710b262fc6c0 (LWP 2475624)):
#0 0x0000710b28520fbd in ioctl () from /nix/store/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61/lib/libc.so.6
#1 0x00006229574f31cc in dedupe_extents ()
#2 0x00006229574fb9df in dedupe_worker ()
#3 0x0000710b28ae33fa in g_thread_pool_thread_proxy () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#4 0x0000710b28ae2c2e in g_thread_proxy () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#5 0x0000710b2849dd53 in start_thread () from /nix/store/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61/lib/libc.so.6
#6 0x0000710b2852563c in __clone3 () from /nix/store/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61/lib/libc.so.6
Thread 1 (Thread 0x710b283deb00 (LWP 607042)):
#0 0x0000710b2852320d in syscall () from /nix/store/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61/lib/libc.so.6
#1 0x0000710b28ae2457 in g_cond_wait () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#2 0x0000710b28ae3d6e in g_thread_pool_free () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#3 0x00006229574fc203 in dedupe_results ()
#4 0x00006229574eb3e9 in main ()
Activity
dedupe: stop instead of spinning when a round makes no progress (mark…
Merge pull request #27 from martinus/fix/dedupe-zero-progress
docs: update upstream-issue status after the fix PRs
Saroumane commentedon Aug 5, 2026
Hello @JackSlateur @trofi , I assume you had the courage to take over the maintenace of duperemove from @markfasheh , thanks for that !!
It seems there are 13 PR waiting to be reviewed / merged (most from @martinus , some of them maybe fixing hangs / loops). Would you mind taking care of them if you have the time ?
Again, thanks a lot.
martinus commentedon Aug 6, 2026
Note that I've created the PRs that I've filled while working on my fork https://github.com/martinus/oans
Which has a lot more correctness and performance work, but I tried to backport all least the critical fixes.