How newren/git-filter-repo works
Project Type and Purpose
• Type: Open-source CLI tool + library.
• Purpose: Rewriting Git repository history at scale via a Python-based library, driving Git’s fast-export/fast-import.
Key Architectural Components
a. User & CLI Layer
– git-filter-repo (single‐file Python script
• Entry point on $PATH
• Parses CLI flags, loads pyproject.toml metadata
– contrib/filter-repo-demos
• Example extensions & custom scripts demonstrating use of the library.
b. Core Library Layer
– git_filter_repo.py
• Implements streaming pipeline:
1. Invocation of “git fast-export”
2. Parsing of commits, trees, blobs
3. Filter callbacks & transform logic
4. Pruning/renaming operations
5. Generation of new stream for “git fast-import”
– Plugin API
• User-provided Python callbacks (via filters, path selectors, mailmap handlers)
c. External Dependencies
– Git (≥2.36.0)
• fast-export, fast-import, update-ref, gc
– Python 3.6+
– CI / Automation
• .github/workflows/test.yml: runs tests on each push/PR
d. Test & Validation Layer
– t/ directory
• Shell scripts (t9390-t9394) and helper scripts for run_tests, run_coverage
• Test-library functions (test-lib.sh, test-lib-functions.sh)
• Test data subdirs for edge cases (empty commits, unusual filenames, mailmap, merges)
e. Documentation Layer
– Documentation/
• Contributing.md, FAQ.md, examples, conversion cheat‐sheets
– README.md, INSTALL.md
Relationships & Data Flows
• User invokes git-filter-repo → CLI script
→ Core library invokes Git fast-export (stream out old history)
→ Streaming parser breaks input into objects
→ Filter pipeline applies “keep vs remove”, “rename”, “prune empty/degenerate”
→ Modified stream fed into Git fast-import → Writes rewritten history
→ Post-import: update-ref, git gc
• CLI can load contrib/demo scripts → Extends core library at filter callbacks
• Tests exercise CLI + library → verify intended behavior on sample repos
• CI workflow → triggers tests on GitHub
Architectural Patterns & Principles
• Layered design: CLI → Library → External Git
• Streaming pipeline: handles arbitrarily large repos without full in-memory load
• Plugin/extensibility API: users supply Python callbacks rather than shell hacks
• Safety by design: fresh-clone enforcement, automatic pruning & repacking
• Single-file CLI distribution for ease of install
Detail Checklist
– Show How CLI flags map into library options
– Show pruning & renaming sub-components within filter pipeline
– Illustrate test directories invoking CLI with sample repos
– Indicate installation path (pyproject.toml, Makefile, single‐file distribution)
– Captures all critical layers (User/CLI, Core Library, Git, Tests, CI, Docs)
– Shows precise data flows and extension points
– Reflects the project’s streaming, plugin-based architecture