# tdurieux/anonymous_github architecture

> Architecture diagram of the GitHub repository tdurieux/anonymous_github, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tdurieux/anonymous_github
- Repository: https://github.com/tdurieux/anonymous_github

## Overview

##### Project type and purpose
- This is a full-stack, TypeScript-based web application with an accompanying CLI tool.
- Purpose: download or proxy GitHub repositories, anonymize metadata and content for double-blind review workflows, cache both original and anonymized artifacts, and expose them via a web UI or CLI-generated ZIP.

##### File structure
- Top-level:
- public/ – static frontend assets (HTML, CSS, JS, Ace editor library, i18n, images, partials)
- src/
◦ cli/ – command-line interface entrypoint
◦ core/ – business logic: anonymization, GitHub sources, models, storage adapters
◦ server/ – Express HTTP server: routes, database connection, scheduled jobs
◦ queue/ – background task definitions (download, cleanup)
◦ streamer/ – streaming endpoint for large file responses
- config/build: Dockerfile, docker-compose.yml, OpenTelemetry and Prometheus configs, gulpfile.js
- package.json & tsconfig.json
- Models under src/core/model use Mongoose schemas (MongoDB).
- Storage abstraction supports local filesystem or S3 (src/core/storage).
- queue/index.ts wires up workers that process tasks asynchronously.

##### README
- Confirms architecture: web server, CLI, caching layer, anonymization engine, storage, and GitHub as an external service.
- Deployment via Docker Compose, environment variables, behind an nginx proxy.

##### Main components
a. Client layer
- Browser (frontend single-page app)
- CLI tool (local node process)
b. API & Application server
- Express server (src/server) exposing REST endpoints for:
- Authentication (GitHub OAuth)
- Repository, PullRequest, Conference management
- File streaming (via src/streamer)
c. Background processing
- Queue system & Workers (src/queue + src/core/source + src/core/anonymize-utils)
d. Data stores
- MongoDB (Mongoose models in src/core/model)
- File storage: Local FS or S3 adapter (src/core/storage)
e. External dependencies
- GitHub API
- (Optionally) AWS S3
f. Observability & Monitoring
- OpenTelemetry collector
- Prometheus scraping metrics (prometheus.yaml)
g. Infrastructure orchestration
- Docker Compose network: server, database, (maybe Redis or queue broker), OTEL collector, Prometheus

##### Relationships and data flows
- Browser → HTTPS → Express server → MongoDB (for metadata)
- Browser → Express server → Streamer → File storage (caches/anonymized files)
- CLI → Core modules (direct library calls to GitHub, anonymization, zip creation)
- Express server → enqueues jobs → Queue broker → Workers pick up →
- Fetch from GitHub API (src/core/source)
- Anonymize (src/core/anonymize-utils)
- Persist anonymized artifacts to storage & update MongoDB
- Metric emission (HTTP server, workers) → OpenTelemetry → Collector → Prometheus

##### Architectural patterns and design principles
- Layered architecture: presentation (public/), application (server), domain (core), infrastructure (storage, queue)
- Adapter pattern for storage (FileSystem / S3) and source (GitHub / Zip)
- Asynchronous worker pattern for long-running tasks
- MVC-like routing in Express with route-utils
- Single codebase handling multiple deployment modes (web + CLI)

##### What
- Notes on deployment (Docker Compose overlay connecting containers)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        BrowserUI["Browser UI\n(SPA)"]:::frontend
        CLI["CLI Tool\n(Local Node)"]:::cli
    end

    click BrowserUI "https://github.com/tdurieux/anonymous_github/tree/main/public/"
    click CLI "https://github.com/tdurieux/anonymous_github/blob/main/src/cli/index.ts"

    %% Application Layer
    subgraph "Application Layer"
        direction TB
        APIServer["Express API Server"]:::backend
        subgraph "Server Components"
            direction TB
            Routes["Routes (MVC)"]:::backend
            DBConn["Database Connection\n(MongoDB/Mongoose)"]:::database
            Scheduler["Scheduled Jobs"]:::backend
            Streamer["File Streaming\nEndpoint"]:::backend
        end
    end

    click APIServer "https://github.com/tdurieux/anonymous_github/blob/main/src/server/index.ts"
    click Routes "https://github.com/tdurieux/anonymous_github/tree/main/src/server/routes/"
    click DBConn "https://github.com/tdurieux/anonymous_github/blob/main/src/server/database.ts"
    click Scheduler "https://github.com/tdurieux/anonymous_github/blob/main/src/server/schedule.ts"
    click Streamer "https://github.com/tdurieux/anonymous_github/blob/main/src/streamer/index.ts"
    click Streamer "https://github.com/tdurieux/anonymous_github/blob/main/src/streamer/route.ts"

    %% Domain & Worker Layer
    subgraph "Domain & Worker Layer"
        direction TB
        QueueBroker["Queue Broker & Worker Pool"]:::worker
        Anonymizer["Anonymization Engine"]:::domain
        SourceAdapters["GitHub Source Adapters"]:::domain
        ZipAdapter["ZIP Source Adapter"]:::domain
        StorageAdapter["Storage Abstraction"]:::infrastructure
        Models["Domain Models\n(Mongoose Schemas)"]:::database
    end

    click QueueBroker "https://github.com/tdurieux/anonymous_github/blob/main/src/queue/index.ts"
    click QueueBroker "https://github.com/tdurieux/anonymous_github/tree/main/src/queue/processes/"
    click Anonymizer "https://github.com/tdurieux/anonymous_github/blob/main/src/core/anonymize-utils.ts"
    click SourceAdapters "https://github.com/tdurieux/anonymous_github/tree/main/src/core/source/"
    click ZipAdapter "https://github.com/tdurieux/anonymous_github/blob/main/src/core/source/Zip.ts"
    click StorageAdapter "https://github.com/tdurieux/anonymous_github/blob/main/src/core/storage/Storage.ts"
    click StorageAdapter "https://github.com/tdurieux/anonymous_github/blob/main/src/core/storage/FileSystem.ts"
    click StorageAdapter "https://github.com/tdurieux/anonymous_github/blob/main/src/core/storage/S3.ts"
    click Models "https://github.com/tdurieux/anonymous_github/tree/main/src/core/model/"

    %% Data Stores
    subgraph "Data Stores"
        direction TB
        MongoDB[(MongoDB)]:::database
        FileStorage[(File Storage\nLocal FS / S3)]:::storage
    end

    %% External Services
    subgraph "External Services"
        direction TB
        GitHubAPI[(GitHub API)]:::external
        S3[(AWS S3)]:::external
    end

    %% Monitoring
    subgraph "Monitoring & Observability"
        direction TB
        OTEL["OTEL Collector"]:::monitoring
        Prometheus["Prometheus"]:::monitoring
    end

    %% Infrastructure
    subgraph "Infrastructure / Deployment"
        direction TB
        DockerCompose["Docker Compose"]:::infra
        Dockerfile["Dockerfile"]:::infra
        Gulpfile["gulpfile.js"]:::infra
    end

    %% Data Flows
    BrowserUI -->|"HTTPS"| APIServer
    CLI -->|"Library Calls"| Anonymizer
    CLI --> Anonymizer
    CLI --> ZipAdapter

    APIServer -->|"Mongoose"| MongoDB
    APIServer -->|"REST/Queue"| QueueBroker
    APIServer -->|"Stream Files"| Streamer

    QueueBroker -->|"Processes Tasks"| Anonymizer
    Anonymizer -->|"Uses Adapters"| SourceAdapters
    Anonymizer -->|"Uses Adapter"| ZipAdapter
    Anonymizer -->|"Writes"| FileStorage
    Anonymizer -->|"Updates"| MongoDB

    SourceAdapters -->|"Fetch Repo"| GitHubAPI
    StorageAdapter -->|"SDK"| FileStorage

    APIServer -->|"Metrics OTLP"| OTEL
    QueueBroker -->|"Metrics OTLP"| OTEL
    Anonymizer -->|"Metrics OTLP"| OTEL
    OTEL -->|"Scrape"| Prometheus

    %% Deployment Connections
    DockerCompose --- APIServer
    DockerCompose --- MongoDB
    DockerCompose --- QueueBroker
    DockerCompose --- OTEL
    DockerCompose --- Prometheus

    %% Styles
    classDef frontend fill:#B3E5FC,stroke:#0288D1,color:#000
    classDef cli fill:#E1BEE7,stroke:#7B1FA2,color:#000
    classDef backend fill:#C8E6C9,stroke:#2E7D32,color:#000
    classDef worker fill:#FFECB3,stroke:#F9A825,color:#000
    classDef domain fill:#FFE0B2,stroke:#EF6C00,color:#000
    classDef database fill:#D1C4E9,stroke:#5E35B1,color:#000
    classDef storage fill:#B2DFDB,stroke:#00796B,color:#000
    classDef external fill:#FFCDD2,stroke:#C62828,color:#000
    classDef monitoring fill:#F0F4C3,stroke:#AFB42B,color:#000
    classDef infra fill:#CFD8DC,stroke:#455A64,color:#000
```
