# markfasheh/duperemove architecture

> Architecture diagram of the GitHub repository markfasheh/duperemove, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/markfasheh/duperemove
- Repository: https://github.com/markfasheh/duperemove

## Overview

Duperemove scans files, hashes their contents by extent (and optionally by block), finds matching candidates, verifies duplicate extents, and can submit those extents to the kernel for deduplication. Its hashfile lets later runs reuse stored metadata and avoid rehashing unchanged files; fdupes input is another documented way to provide files. The graph reflects the README and sampled implementation excerpts; some orchestration links rely on documented behavior because the main routine is only partially shown.

Main entry points: [`duperemove.c`](https://github.com/markfasheh/duperemove/blob/master/duperemove.c) (CLI orchestration), [`find_dupes.c`](https://github.com/markfasheh/duperemove/blob/master/find_dupes.c) (Duplicate search), [`run_dedupe.c`](https://github.com/markfasheh/duperemove/blob/master/run_dedupe.c) (Dedupe coordination), [`file_scan.c`](https://github.com/markfasheh/duperemove/blob/master/file_scan.c) (File scan and hashing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_input["Input and scan"]
  node_cli["CLI orchestration<br/>[duperemove.c]"]
  node_scan["File scan and hashing<br/>[file_scan.c]"]
  node_checksum["Checksum routines<br/>[csum.c]"]
  node_records["File records<br/>[filerec.c]"]
end

subgraph group_matching["Matching and results"]
  node_hash_tree["Block hash index<br/>[hash-tree.c]"]
  node_search["Duplicate search<br/>[find_dupes.c]"]
  node_results["Duplicate results<br/>[results-tree.c]"]
end

subgraph group_dedupe["Dedupe execution"]
  node_extent_map["Extent mapping<br/>[fiemap.c]"]
  node_dedupe_run["Dedupe coordination<br/>[run_dedupe.c]"]
  node_dedupe_ioctl["Dedupe requests<br/>[dedupe.c]"]
end

subgraph group_state["Persistent state"]
  node_hashdb[("Hashfile database<br/>[dbfile.c]")]
end

node_user(("User"))
node_fdupes["fdupes stream"]
node_files["Input files"]
node_kernel{{"Linux kernel"}}

node_user -->|"provides options"| node_cli
node_files -->|"provides paths"| node_cli
node_fdupes -.->|"streams paths"| node_cli
node_cli -->|"starts scan"| node_scan
node_scan -->|"computes digests"| node_checksum
node_scan -->|"updates records"| node_records
node_scan -->|"stores hashes"| node_hashdb
node_cli -->|"loads hashfile"| node_hashdb
node_cli -->|"builds index"| node_hash_tree
node_cli -->|"starts search"| node_search
node_search -->|"looks up blocks"| node_hash_tree
node_search -->|"loads extents"| node_hashdb
node_search -->|"verifies matches"| node_checksum
node_search -->|"adds matches"| node_results
node_cli -->|"starts dedupe"| node_dedupe_run
node_dedupe_run -->|"reads candidates"| node_results
node_dedupe_run -->|"checks mappings"| node_extent_map
node_dedupe_run -->|"dispatches requests"| node_dedupe_ioctl
node_dedupe_ioctl -->|"submits ioctl"| node_kernel
node_dedupe_run -->|"removes stale hashes"| node_hashdb

click node_cli "https://github.com/markfasheh/duperemove/blob/master/duperemove.c"
click node_scan "https://github.com/markfasheh/duperemove/blob/master/file_scan.c"
click node_checksum "https://github.com/markfasheh/duperemove/blob/master/csum.c"
click node_records "https://github.com/markfasheh/duperemove/blob/master/filerec.c"
click node_hashdb "https://github.com/markfasheh/duperemove/blob/master/dbfile.c"
click node_hash_tree "https://github.com/markfasheh/duperemove/blob/master/hash-tree.c"
click node_search "https://github.com/markfasheh/duperemove/blob/master/find_dupes.c"
click node_results "https://github.com/markfasheh/duperemove/blob/master/results-tree.c"
click node_extent_map "https://github.com/markfasheh/duperemove/blob/master/fiemap.c"
click node_dedupe_run "https://github.com/markfasheh/duperemove/blob/master/run_dedupe.c"
click node_dedupe_ioctl "https://github.com/markfasheh/duperemove/blob/master/dedupe.c"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_scan,node_checksum,node_records,node_user toneBlue
class node_hash_tree,node_search,node_results toneAmber
class node_extent_map,node_dedupe_run,node_dedupe_ioctl toneMint
class node_hashdb toneRose
class node_fdupes,node_files,node_kernel toneIndigo
```

## Components

### Input and scan

- [CLI orchestration](https://github.com/markfasheh/duperemove/blob/master/duperemove.c): `duperemove.c`
- [File scan and hashing](https://github.com/markfasheh/duperemove/blob/master/file_scan.c): `file_scan.c`
- [Checksum routines](https://github.com/markfasheh/duperemove/blob/master/csum.c): `csum.c`
- [File records](https://github.com/markfasheh/duperemove/blob/master/filerec.c): `filerec.c`

### Matching and results

- [Block hash index](https://github.com/markfasheh/duperemove/blob/master/hash-tree.c): `hash-tree.c`
- [Duplicate search](https://github.com/markfasheh/duperemove/blob/master/find_dupes.c): `find_dupes.c`
- [Duplicate results](https://github.com/markfasheh/duperemove/blob/master/results-tree.c): `results-tree.c`

### Dedupe execution

- [Extent mapping](https://github.com/markfasheh/duperemove/blob/master/fiemap.c): `fiemap.c`
- [Dedupe coordination](https://github.com/markfasheh/duperemove/blob/master/run_dedupe.c): `run_dedupe.c`
- [Dedupe requests](https://github.com/markfasheh/duperemove/blob/master/dedupe.c): `dedupe.c`

### Persistent state

- [Hashfile database](https://github.com/markfasheh/duperemove/blob/master/dbfile.c): `dbfile.c`

### Other components

- **User**
- **fdupes stream**
- **Input files**
- **Linux kernel**

## Connections

- User → CLI orchestration: provides options
- Input files → CLI orchestration: provides paths
- fdupes stream → CLI orchestration: streams paths
- CLI orchestration → File scan and hashing: starts scan
- File scan and hashing → Checksum routines: computes digests
- File scan and hashing → File records: updates records
- File scan and hashing → Hashfile database: stores hashes
- CLI orchestration → Hashfile database: loads hashfile
- CLI orchestration → Block hash index: builds index
- CLI orchestration → Duplicate search: starts search
- Duplicate search → Block hash index: looks up blocks
- Duplicate search → Hashfile database: loads extents
- Duplicate search → Checksum routines: verifies matches
- Duplicate search → Duplicate results: adds matches
- CLI orchestration → Dedupe coordination: starts dedupe
- Dedupe coordination → Duplicate results: reads candidates
- Dedupe coordination → Extent mapping: checks mappings
- Dedupe coordination → Dedupe requests: dispatches requests
- Dedupe requests → Linux kernel: submits ioctl
- Dedupe coordination → Hashfile database: removes stale hashes