How arthaud/git-dumper works
Main Components
a. CLI Interface
b. Core Orchestrator
c. HTTP Client Layer
d. Directory Listing Checker
e. Smart Fetch Modules
• Common Files Fetcher
• Refs Resolver
• Object Resolver
• Recursive Object Fetcher
f. Concurrency / Request Manager
g. Filesystem Writer & Git Checkout
h. Configuration & Extensions (proxy, headers, certs)
Relationships & Data Flows
• User → CLI Interface: parses arguments (URL, DIR, options).
• CLI Interface → Core Orchestrator: kicks off dump process.
• Core Orchestrator ↔ Concurrency Manager: schedules HTTP requests.
• Core Orchestrator → Directory Listing Checker: probes for “Index of /.git/”.
– If listing available → HTTP Client Layer → Filesystem Writer (bulk download).
– Else → engage Smart Fetch Modules:
· Common Files Fetcher uses HTTP Client → retrieves .git HEAD, index, config.
· Refs Resolver consumes fetched data → builds list of ref paths.
· Object Resolver scans refs & index → builds list of object SHA1 URLs.
· Recursive Object Fetcher pulls each object, analyzes commits → discovers parent objects (loop).
• Each HTTP fetch → HTTP Client Layer → returns bytes → passed back to orchestrator → Filesystem Writer.
• After downloading all objects → Filesystem Writer writes to local .git directory → invokes subprocess “git checkout .”
Architectural Patterns & Design Principles
– Modular separation: networking, parsing, I/O, orchestration.
– Single responsibility: each fetch module focuses on one aspect of the Git data model.
– Pluggable HTTP settings: proxy, timeout, headers, client cert.
– Concurrency abstraction: can tune number of simultaneous requests (–jobs).
Key Technologies & Libraries
– Python 3 (pyproject.toml / setup.cfg)
– requests (or urllib3) for HTTP operations
– argparse / click for CLI parsing
– threading / asyncio for concurrency
– subprocess for invoking git checkout
– standard os/path libraries for filesystem I/O
Detailed Component Breakdown
[1] CLI Interface
• Parses URL, DIR,FLAGS → hands off to Orchestrator
[2] Core Orchestrator
• Coordinates overall workflow
• Decides directory listing vs smart fetch
[3] Directory Listing Checker
• HEAD/GET /.git/ → HTTP Client
[4] Smart Fetch Modules
a. Common Files Fetcher → fetch HEAD, index, config, packed-refs
b. Refs Resolver → parse fetched files → list ref URLs
c. Object Resolver → scan refs + index → list object URLs
d. Recursive Object Fetcher → BFS/DFS fetch of object graph
[5] Concurrency / Request Manager
• ThreadPool or Async IO
• Manages retries, timeouts, user-agent, proxy
[6] HTTP Client Layer
• Encapsulates requests library
• Applies global options (–proxy, –timeout, –header, –cert)
[7] Filesystem Writer & Git Checkout
• Writes raw objects to .git/objects/…
• Copies refs to .git/refs/…
• subprocess.run([“git”, “checkout”, “.”])
[8] Configuration & Extensions
• requirements.txt / pyproject.toml define dependencies
• setup.cfg configures linting, packaging
– Notes alternative flows (directory listing vs smart fetch).
– Highlights concurrency and configuration extension points.