# newren/git-filter-repo architecture

> Architecture diagram of the GitHub repository newren/git-filter-repo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/newren/git-filter-repo
- Repository: https://github.com/newren/git-filter-repo

## Overview

##### Project Type and Purpose
- Type: Open-source CLI tool + library.
- Purpose: Rewriting Git repository history at scale via a Python-based library, driving Git’s fast-export/fast-import.

##### Key Architectural Components
a. User & CLI Layer
– git-filter-repo (single‐file Python script
- Entry point on $PATH
- Parses CLI flags, loads pyproject.toml metadata
– contrib/filter-repo-demos
- Example extensions & custom scripts demonstrating use of the library.
b. Core Library Layer
– git_filter_repo.py
- Implements streaming pipeline:
1. Invocation of “git fast-export”
2. Parsing of commits, trees, blobs
3. Filter callbacks & transform logic
4. Pruning/renaming operations
5. Generation of new stream for “git fast-import”
– Plugin API
- User-provided Python callbacks (via filters, path selectors, mailmap handlers)
c. External Dependencies
– Git (≥2.36.0)
- fast-export, fast-import, update-ref, gc
– Python 3.6+
– CI / Automation
- .github/workflows/test.yml: runs tests on each push/PR
d. Test & Validation Layer
– t/ directory
- Shell scripts (t9390-t9394) and helper scripts for run_tests, run_coverage
- Test-library functions (test-lib.sh, test-lib-functions.sh)
- Test data subdirs for edge cases (empty commits, unusual filenames, mailmap, merges)
e. Documentation Layer
– Documentation/
- Contributing.md, FAQ.md, examples, conversion cheat‐sheets
– README.md, INSTALL.md

##### Relationships & Data Flows
- User invokes git-filter-repo → CLI script
→ Core library invokes Git fast-export (stream out old history)
→ Streaming parser breaks input into objects
→ Filter pipeline applies “keep vs remove”, “rename”, “prune empty/degenerate”
→ Modified stream fed into Git fast-import → Writes rewritten history
→ Post-import: update-ref, git gc
- CLI can load contrib/demo scripts → Extends core library at filter callbacks
- Tests exercise CLI + library → verify intended behavior on sample repos
- CI workflow → triggers tests on GitHub

##### Architectural Patterns & Principles
- Layered design: CLI → Library → External Git
- Streaming pipeline: handles arbitrarily large repos without full in-memory load
- Plugin/extensibility API: users supply Python callbacks rather than shell hacks
- Safety by design: fresh-clone enforcement, automatic pruning & repacking
- Single-file CLI distribution for ease of install

##### Detail Checklist
– Show How CLI flags map into library options
– Show pruning & renaming sub-components within filter pipeline
– Illustrate test directories invoking CLI with sample repos
– Indicate installation path (pyproject.toml, Makefile, single‐file distribution)

– Captures all critical layers (User/CLI, Core Library, Git, Tests, CI, Docs)
– Shows precise data flows and extension points
– Reflects the project’s streaming, plugin-based architecture

## Diagram (Mermaid)

```mermaid
graph TD
    %% User & CLI Layer
    subgraph "User & CLI Layer"
        CLI["git-filter-repo CLI"]:::internal
        PYPROJECT["pyproject.toml"]:::internal
        MAKEFILE["Makefile"]:::internal
    end
    click CLI "https://github.com/newren/git-filter-repo/tree/main/git-filter-repo"
    click PYPROJECT "https://github.com/newren/git-filter-repo/blob/main/pyproject.toml"
    click MAKEFILE "https://github.com/newren/git-filter-repo/tree/main/Makefile"

    %% Core Library Layer
    subgraph "Core Library Layer"
        CORE["git_filter_repo.py\n(Stream Pipeline)"]:::internal
        subgraph "Filter Pipeline"
            EXPORT["git fast-export\n(stdin)"]:::external
            PARSER["Parser (commits, trees, blobs)"]:::internal
            CALLBACKS["Filter Callbacks"]:::internal
            PRUNE["Prune & Rename Operations"]:::internal
            IMPORT["git fast-import\n(stdout)"]:::external
        end
    end
    click CORE "https://github.com/newren/git-filter-repo/blob/main/git_filter_repo.py"

    %% External Dependencies
    subgraph "External Dependencies"
        GIT["Git Commands\n(fast-export, fast-import,\nupdate-ref, gc)"]:::external
    end

    %% Plugin & Extensions
    subgraph "Plugin & Extensions"
        DEMOS["contrib/filter-repo-demos"]:::extension
        DEMO_README["README.md"]:::extension
    end
    click DEMOS "https://github.com/newren/git-filter-repo/tree/main/contrib/filter-repo-demos/"
    click DEMO_README "https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-demos/README.md"

    %% Test & Validation Layer
    subgraph "Test & Validation Layer"
        RUN_TESTS["t/run_tests"]:::tests
        RUN_COV["t/run_coverage"]:::tests
        TEST_BASIC["t/t9390-filter-repo-basics.sh"]:::tests
        DATA9390["t/t9390/"]:::tests
        TEST_LIB["t/test-lib.sh\n& test-lib-functions.sh"]:::tests
        TEST_LIB_USAGE["t/t9391-filter-repo-lib-usage.sh"]:::tests
        DATA9391["t/t9391/"]:::tests
        TEST_PYCB["t/t9392-filter-repo-python-callback.sh"]:::tests
        RERUN["t/t9393-filter-repo-rerun.sh"]:::tests
        DATA9393["t/t9393/"]:::tests
        TEST_SANITY["t/t9394-filter-repo-sanity-checks-and-bigger-repo-setup.sh"]:::tests
        DATA9394["t/t9394/"]:::tests
    end
    click RUN_TESTS "https://github.com/newren/git-filter-repo/tree/main/t/run_tests"
    click RUN_COV "https://github.com/newren/git-filter-repo/tree/main/t/run_coverage"
    click TEST_BASIC "https://github.com/newren/git-filter-repo/blob/main/t/t9390-filter-repo-basics.sh"
    click DATA9390 "https://github.com/newren/git-filter-repo/tree/main/t/t9390/"
    click TEST_LIB "https://github.com/newren/git-filter-repo/blob/main/t/test-lib.sh"
    click TEST_LIB "https://github.com/newren/git-filter-repo/blob/main/t/test-lib-functions.sh"
    click TEST_LIB_USAGE "https://github.com/newren/git-filter-repo/blob/main/t/t9391-filter-repo-lib-usage.sh"
    click DATA9391 "https://github.com/newren/git-filter-repo/tree/main/t/t9391/"
    click TEST_PYCB "https://github.com/newren/git-filter-repo/blob/main/t/t9392-filter-repo-python-callback.sh"
    click RERUN "https://github.com/newren/git-filter-repo/blob/main/t/t9393-filter-repo-rerun.sh"
    click DATA9393 "https://github.com/newren/git-filter-repo/tree/main/t/t9393/"
    click TEST_SANITY "https://github.com/newren/git-filter-repo/blob/main/t/t9394-filter-repo-sanity-checks-and-bigger-repo-setup.sh"
    click DATA9394 "https://github.com/newren/git-filter-repo/tree/main/t/t9394/"

    %% CI / Automation
    subgraph "CI / Automation"
        WORKFLOW[".github/workflows/test.yml"]:::ci
    end
    click WORKFLOW "https://github.com/newren/git-filter-repo/blob/main/.github/workflows/test.yml"

    %% Documentation Layer
    subgraph "Documentation"
        DOC1["Documentation/git-filter-repo.txt"]:::docs
        DOC2["Documentation/Contributing.md"]:::docs
        DOC3["Documentation/FAQ.md"]:::docs
        DOC4["Documentation/converting-from-filter-branch.md"]:::docs
        DOC5["Documentation/converting-from-bfg-repo-cleaner.md"]:::docs
        DOC6["Documentation/examples-from-user-filed-issues.md"]:::docs
        README["README.md"]:::docs
        INSTALL["INSTALL.md"]:::docs
    end
    click DOC1 "https://github.com/newren/git-filter-repo/blob/main/Documentation/git-filter-repo.txt"
    click DOC2 "https://github.com/newren/git-filter-repo/blob/main/Documentation/Contributing.md"
    click DOC3 "https://github.com/newren/git-filter-repo/blob/main/Documentation/FAQ.md"
    click DOC4 "https://github.com/newren/git-filter-repo/blob/main/Documentation/converting-from-filter-branch.md"
    click DOC5 "https://github.com/newren/git-filter-repo/blob/main/Documentation/converting-from-bfg-repo-cleaner.md"
    click DOC6 "https://github.com/newren/git-filter-repo/blob/main/Documentation/examples-from-user-filed-issues.md"
    click README "https://github.com/newren/git-filter-repo/blob/main/README.md"
    click INSTALL "https://github.com/newren/git-filter-repo/blob/main/INSTALL.md"

    %% Data Flows & Relationships
    CLI -->|parses flags| CORE
    CLI -->|reads| PYPROJECT
    CORE -->|invokes| EXPORT
    EXPORT -->|streams| PARSER
    PARSER -->|to| CALLBACKS
    CALLBACKS -->|to| PRUNE
    PRUNE -->|to| IMPORT
    IMPORT -->|writes| GIT
    GIT -->|post-import| CORE
    CALLBACKS -->|hooks| DEMOS
    DEMOS -->|examples| CORE

    RUN_TESTS -->|executes CLI| CLI
    RUN_COV -->|executes CLI| CLI
    TEST_BASIC --> CLI
    TEST_LIB_USAGE --> CLI
    TEST_PYCB --> CLI
    RERUN --> CLI
    TEST_SANITY --> CLI

    WORKFLOW -->|triggers| RUN_TESTS
    WORKFLOW -->|triggers| RUN_COV

    %% Styles
    classDef internal fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef external fill:#d4edda,stroke:#333,stroke-width:1px
    classDef extension fill:#ffe5cc,stroke:#333,stroke-width:1px
    classDef docs fill:#e2e3e5,stroke:#333,stroke-width:1px
    classDef tests fill:#f5e6ff,stroke:#333,stroke-width:1px
    classDef ci fill:#f0f0f0,stroke:#333,stroke-width:1px
```
