How markfasheh/duperemove works
Duperemove scans files, hashes their contents by extent (and optionally by block), finds matching candidates, verifies duplicate extents, and can submit those extents to the kernel for deduplication. Its hashfile lets later runs reuse stored metadata and avoid rehashing unchanged files; fdupes input is another documented way to provide files. The graph reflects the README and sampled implementation excerpts; some orchestration links rely on documented behavior because the main routine is only partially shown.
Main entry points: duperemove.c CLI orchestration · find_dupes.c Duplicate search · run_dedupe.c Dedupe coordination · file_scan.c File scan and hashing
Components 15
Input and scan
- CLI orchestration
duperemove.c - File scan and hashing
file_scan.c - Checksum routines
csum.c - File records
filerec.c
Matching and results
- Block hash index
hash-tree.c - Duplicate search
find_dupes.c - Duplicate results
results-tree.c
Dedupe execution
- Extent mapping
fiemap.c - Dedupe coordination
run_dedupe.c - Dedupe requests
dedupe.c
Persistent state
- Hashfile database
dbfile.c
Other components
- User
- fdupes stream
- Input files
- Linux kernel
Connections 20
- User → CLI orchestration: provides options
- Input files → CLI orchestration: provides paths
- fdupes stream → CLI orchestration: streams paths
- CLI orchestration → File scan and hashing: starts scan
- File scan and hashing → Checksum routines: computes digests
- File scan and hashing → File records: updates records
- File scan and hashing → Hashfile database: stores hashes
- CLI orchestration → Hashfile database: loads hashfile
- CLI orchestration → Block hash index: builds index
- CLI orchestration → Duplicate search: starts search
- Duplicate search → Block hash index: looks up blocks
- Duplicate search → Hashfile database: loads extents
- Duplicate search → Checksum routines: verifies matches
- Duplicate search → Duplicate results: adds matches
- CLI orchestration → Dedupe coordination: starts dedupe
- Dedupe coordination → Duplicate results: reads candidates
- Dedupe coordination → Extent mapping: checks mappings
- Dedupe coordination → Dedupe requests: dispatches requests
- Dedupe requests → Linux kernel: submits ioctl
- Dedupe coordination → Hashfile database: removes stale hashes