How tdurieux/anonymous_github works
Project type and purpose
• This is a full-stack, TypeScript-based web application with an accompanying CLI tool.
• Purpose: download or proxy GitHub repositories, anonymize metadata and content for double-blind review workflows, cache both original and anonymized artifacts, and expose them via a web UI or CLI-generated ZIP.
File structure
• Top-level:
• public/ – static frontend assets (HTML, CSS, JS, Ace editor library, i18n, images, partials)
• src/
◦ cli/ – command-line interface entrypoint
◦ core/ – business logic: anonymization, GitHub sources, models, storage adapters
◦ server/ – Express HTTP server: routes, database connection, scheduled jobs
◦ queue/ – background task definitions (download, cleanup)
◦ streamer/ – streaming endpoint for large file responses
• config/build: Dockerfile, docker-compose.yml, OpenTelemetry and Prometheus configs, gulpfile.js
• package.json & tsconfig.json
• Models under src/core/model use Mongoose schemas (MongoDB).
• Storage abstraction supports local filesystem or S3 (src/core/storage).
• queue/index.ts wires up workers that process tasks asynchronously.
README
• Confirms architecture: web server, CLI, caching layer, anonymization engine, storage, and GitHub as an external service.
• Deployment via Docker Compose, environment variables, behind an nginx proxy.
Main components
a. Client layer
• Browser (frontend single-page app)
• CLI tool (local node process)
b. API & Application server
• Express server (src/server) exposing REST endpoints for:
• Authentication (GitHub OAuth)
• Repository, PullRequest, Conference management
• File streaming (via src/streamer)
c. Background processing
• Queue system & Workers (src/queue + src/core/source + src/core/anonymize-utils)
d. Data stores
• MongoDB (Mongoose models in src/core/model)
• File storage: Local FS or S3 adapter (src/core/storage)
e. External dependencies
• GitHub API
• (Optionally) AWS S3
f. Observability & Monitoring
• OpenTelemetry collector
• Prometheus scraping metrics (prometheus.yaml)
g. Infrastructure orchestration
• Docker Compose network: server, database, (maybe Redis or queue broker), OTEL collector, Prometheus
Relationships and data flows
• Browser → HTTPS → Express server → MongoDB (for metadata)
• Browser → Express server → Streamer → File storage (caches/anonymized files)
• CLI → Core modules (direct library calls to GitHub, anonymization, zip creation)
• Express server → enqueues jobs → Queue broker → Workers pick up →
• Fetch from GitHub API (src/core/source)
• Anonymize (src/core/anonymize-utils)
• Persist anonymized artifacts to storage & update MongoDB
• Metric emission (HTTP server, workers) → OpenTelemetry → Collector → Prometheus
Architectural patterns and design principles
• Layered architecture: presentation (public/), application (server), domain (core), infrastructure (storage, queue)
• Adapter pattern for storage (FileSystem / S3) and source (GitHub / Zip)
• Asynchronous worker pattern for long-running tasks
• MVC-like routing in Express with route-utils
• Single codebase handling multiple deployment modes (web + CLI)
What
• Notes on deployment (Docker Compose overlay connecting containers)