# lakshmipathi/dduper architecture

> Architecture diagram of the GitHub repository lakshmipathi/dduper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lakshmipathi/dduper
- Repository: https://github.com/lakshmipathi/dduper

## Overview

##### Main Components

a) Core Components:
- BTRFS Filesystem Interface
- Checksum Management System
- Deduplication Engine
- File/Directory Scanner
- Validation System

b) Operating Modes:
- Default Mode (Safe Mode)
- Fast Mode
- Insane Mode
- Analysis Mode

##### Detailed Component Breakdown

Input Handler:
**[CLI Interface] → [Input Validator]**
- ↓
**[Path Scanner] → [File/Directory Processor]**

Core Engine:
**[Chunk Size Manager] → [BTRFS Csum Reader]**
- ↓
**[Deduplication Engine]**
- ↓            ↓
**[ficlonerange] [fideduperange]**

Safety System:
**[Backup Manager] → [SHA256 Validator]**
- ↓
**[Recovery Logger]**

##### Integration Points

Highlight:
- BTRFS filesystem hooks
- Kernel system calls
- Checksum verification points
- File operation interfaces

##### Technical Details to Note

- Supported checksum types (crc32, xxhash64, blake2, sha256)
- Backup file paths
- System call integration points
- Validation flow paths

##### Limitations Section

- Subvolume limitations
- Single file block limitations
- Checksum type restrictions

Remember to emphasize the three operating modes (default, fast, and insane) as they represent different trade-offs between safety and performance, which is a key architectural decision in this system.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph CLI
        UI[User Interface]
        InputVal[Input Validator]
        UI --> InputVal
    end

    subgraph Scanner
        PathScan[Path Scanner]
        FileProc["File/Directory Processor"]:::scanner
        PathScan --> FileProc
    end

    subgraph "Core Engine"
        ChunkMgr[Chunk Size Manager]
        CsumRead["BTRFS Csum Reader"]:::checksum
        DedupEng["Deduplication Engine"]:::core
        FiClone["ficlonerange"]
        FiDedup["fideduperange"]
        
        ChunkMgr --> CsumRead
        CsumRead --> DedupEng
        DedupEng --> FiClone
        DedupEng --> FiDedup
    end

    subgraph "Operating Modes"
        DefMode[Default Mode]
        FastMode["Fast Mode"]:::fast
        InsaneMode[Insane Mode]
        AnalMode[Analysis Mode]
    end

    subgraph "Safety System"
        BackupMgr[Backup Manager]
        SHA256[SHA256 Validator]
        RecovLog[Recovery Logger]
        TestVal["Validation System"]:::validation
        
        BackupMgr --> SHA256
        SHA256 --> RecovLog
        RecovLog --> TestVal
    end

    subgraph "BTRFS Integration"
        BTRFSInt["BTRFS Filesystem Interface"]:::btrfs
        KernelInt[Kernel Integration]
        CsumTree[Checksum Tree]
        
        BTRFSInt --> KernelInt
        KernelInt --> CsumTree
    end

    InputVal --> PathScan
    FileProc --> ChunkMgr
    DedupEng --> BTRFSInt
    DefMode & FastMode & InsaneMode & AnalMode --> DedupEng

    classDef core fill:#f9f,stroke:#333,stroke-width:2px
    classDef btrfs fill:#ffa500,stroke:#333
    classDef checksum fill:#90EE90,stroke:#333
    classDef validation fill:#ff6666,stroke:#333
    classDef scanner fill:#87CEEB,stroke:#333
    classDef fast fill:#98FB98,stroke:#333

    click DedupEng "https://github.com/Lakshmipathi/dduper/tree/master/dduper"
    click BTRFSInt "https://github.com/Lakshmipathi/dduper/blob/master/bin/btrfs.static"
    click CsumRead "https://github.com/Lakshmipathi/dduper/blob/master/patch/btrfs-progs-v6.11/0001-Print-csum-for-a-given-file-on-stdout.patch"
    click FileProc "https://github.com/Lakshmipathi/dduper/blob/master/ci/gitlab/tests/basic_dir_recurse_test.sh"
    click TestVal "https://github.com/Lakshmipathi/dduper/blob/master/tests/verify.sh"
    click FastMode "https://github.com/Lakshmipathi/dduper/blob/master/ci/gitlab/tests/fast_mode.sh"
    click SHA256 "https://github.com/Lakshmipathi/dduper/blob/master/ci/gitlab/tests/basic_sanity_csum.sh"
```
