# arthaud/git-dumper architecture

> Architecture diagram of the GitHub repository arthaud/git-dumper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arthaud/git-dumper
- Repository: https://github.com/arthaud/git-dumper

## Overview

##### Main Components
a. CLI Interface
b. Core Orchestrator
c. HTTP Client Layer
d. Directory Listing Checker
e. Smart Fetch Modules
- Common Files Fetcher
- Refs Resolver
- Object Resolver
- Recursive Object Fetcher
f. Concurrency / Request Manager
g. Filesystem Writer & Git Checkout
h. Configuration & Extensions (proxy, headers, certs)

##### Relationships & Data Flows
- User → CLI Interface: parses arguments (URL, DIR, options).
- CLI Interface → Core Orchestrator: kicks off dump process.
- Core Orchestrator ↔ Concurrency Manager: schedules HTTP requests.
- Core Orchestrator → Directory Listing Checker: probes for “Index of /.git/”.
– If listing available → HTTP Client Layer → Filesystem Writer (bulk download).
– Else → engage Smart Fetch Modules:
· Common Files Fetcher uses HTTP Client → retrieves .git HEAD, index, config.
· Refs Resolver consumes fetched data → builds list of ref paths.
· Object Resolver scans refs & index → builds list of object SHA1 URLs.
· Recursive Object Fetcher pulls each object, analyzes commits → discovers parent objects (loop).
- Each HTTP fetch → HTTP Client Layer → returns bytes → passed back to orchestrator → Filesystem Writer.
- After downloading all objects → Filesystem Writer writes to local .git directory → invokes subprocess “git checkout .”

##### Architectural Patterns & Design Principles
– Modular separation: networking, parsing, I/O, orchestration.
– Single responsibility: each fetch module focuses on one aspect of the Git data model.
– Pluggable HTTP settings: proxy, timeout, headers, client cert.
– Concurrency abstraction: can tune number of simultaneous requests (–jobs).

##### Key Technologies & Libraries
– Python 3 (pyproject.toml / setup.cfg)
– requests (or urllib3) for HTTP operations
– argparse / click for CLI parsing
– threading / asyncio for concurrency
– subprocess for invoking git checkout
– standard os/path libraries for filesystem I/O

##### Detailed Component Breakdown
[1] CLI Interface
- Parses URL, DIR,FLAGS → hands off to Orchestrator
[2] Core Orchestrator
- Coordinates overall workflow
- Decides directory listing vs smart fetch
[3] Directory Listing Checker
- HEAD/GET /.git/ → HTTP Client
[4] Smart Fetch Modules
a. Common Files Fetcher → fetch HEAD, index, config, packed-refs
b. Refs Resolver → parse fetched files → list ref URLs
c. Object Resolver → scan refs + index → list object URLs
d. Recursive Object Fetcher → BFS/DFS fetch of object graph
[5] Concurrency / Request Manager
- ThreadPool or Async IO
- Manages retries, timeouts, user-agent, proxy
[6] HTTP Client Layer
- Encapsulates requests library
- Applies global options (–proxy, –timeout, –header, –cert)
[7] Filesystem Writer & Git Checkout
- Writes raw objects to .git/objects/…
- Copies refs to .git/refs/…
- subprocess.run([“git”, “checkout”, “.”])
[8] Configuration & Extensions
- requirements.txt / pyproject.toml define dependencies
- setup.cfg configures linting, packaging

– Notes alternative flows (directory listing vs smart fetch).
– Highlights concurrency and configuration extension points.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Services
    WebServer["Web Server Hosting .git"]:::external
    GitBinary["Local Git Binary"]:::external

    %% Configuration & Extensions
    subgraph "Configuration & Extensions"
        Cfg_pyproject["pyproject.toml"]:::config
        Cfg_requirements["requirements.txt"]:::config
        Cfg_setup["setup.cfg"]:::config
    end

    %% Core Components
    CLI["CLI Interface\n(parses URL, DIR, FLAGS)"]:::internal
    Orch["Core Orchestrator\n(coordinates workflow)"]:::internal
    DirCheck["Directory Listing Checker\n(probes Index of /.git/)"]:::internal
    Concurrency["Concurrency / Request Manager\n(threadpool/asyncio)"]:::internal
    HTTPClient["HTTP Client Layer\n(requests wrapper)"]:::internal

    subgraph "Smart Fetch Modules"
        CF["Common Files Fetcher\n(fetch HEAD, index, config)"]:::internal
        RR["Refs Resolver\n(parse refs)"]:::internal
        OR["Object Resolver\n(list object URLs)"]:::internal
        RF["Recursive Object Fetcher\n(fetch object graph)"]:::internal
    end

    FileWriter["Filesystem Writer & Git Checkout\n(write .git files)"]:::internal
    Storage["Local .git Filesystem"]:::storage

    %% Flows
    CLI -->|"invoke dump() "| Orch
    Orch --> Concurrency
    Orch --> DirCheck

    DirCheck -->|available listing| HTTPClient
    HTTPClient -->|"bulk download"| FileWriter

    DirCheck -.->|"no listing"| CF
    CF -->|"uses HTTP"| HTTPClient
    RR -->|"uses HTTP"| HTTPClient
    OR -->|"uses HTTP"| HTTPClient
    RF -->|"uses HTTP"| HTTPClient

    Orch --> CF
    CF --> RR
    RR --> OR
    OR --> RF

    RF -->|objects data| FileWriter
    FileWriter --> Storage
    FileWriter -->|"git checkout"| GitBinary

    %% Integration with External
    HTTPClient --> WebServer

    %% Configuration usage
    HTTPClient --> Cfg_pyproject
    HTTPClient --> Cfg_requirements
    HTTPClient --> Cfg_setup
    Concurrency --> Cfg_pyproject
    Concurrency --> Cfg_requirements
    Concurrency --> Cfg_setup

    %% Click Events
    click CLI "https://github.com/arthaud/git-dumper/blob/master/git_dumper.py"
    click Orch "https://github.com/arthaud/git-dumper/blob/master/git_dumper.py"
    click DirCheck "https://github.com/arthaud/git-dumper/blob/master/git_dumper.py"
    click CF "https://github.com/arthaud/git-dumper/blob/master/git_dumper.py"
    click RR "https://github.com/arthaud/git-dumper/blob/master/git_dumper.py"
    click OR "https://github.com/arthaud/git-dumper/blob/master/git_dumper.py"
    click RF "https://github.com/arthaud/git-dumper/blob/master/git_dumper.py"
    click Concurrency "https://github.com/arthaud/git-dumper/blob/master/git_dumper.py"
    click HTTPClient "https://github.com/arthaud/git-dumper/blob/master/git_dumper.py"
    click FileWriter "https://github.com/arthaud/git-dumper/blob/master/git_dumper.py"
    click Cfg_pyproject "https://github.com/arthaud/git-dumper/blob/master/pyproject.toml"
    click Cfg_requirements "https://github.com/arthaud/git-dumper/blob/master/requirements.txt"
    click Cfg_setup "https://github.com/arthaud/git-dumper/blob/master/setup.cfg"

    %% Style Definitions
    classDef internal fill:#D0E8FF,stroke:#0366D6,color:#0366D6;
    classDef external fill:#FFE4C4,stroke:#FF8C00,color:#FF8C00;
    classDef storage fill:#DFFFE0,stroke:#28A745,color:#28A745;
    classDef config fill:#F5F5F5,stroke:#999999,color:#555555;
```
