# twitter/the-algorithm-ml architecture

> Architecture diagram of the GitHub repository twitter/the-algorithm-ml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/twitter/the-algorithm-ml
- Repository: https://github.com/twitter/the-algorithm-ml

## Overview

##### Project Type & Purpose
- This is an open-source ML toolkit for training and evaluating two embedding/ranking models.
- It provides shared libraries (“common”, “core”, “metrics”, “ml_logging”, “optimizers”, “reader”), plus two end-to-end projects under `projects/home/recap` and `projects/twhin`.

##### File Structure
- Top-level libraries:
- common (batching, device management, checkpointing, filesystem utils, embedding modules)
- core (configuration, training loops, pipeline, losses, metrics)
- metrics (AUROC, aggregation, RCE)
- ml_logging (absl and torch logging adapters)
- optimizers (configurable optimizer interface)
- reader (data ingestion & utils)
- tools (utilities like PQ)
- Projects:
- home/recap – data generation & preprocessing, model definitions (mask_net, mlp, numeric_calibration), entrypoint, scripts
- twhin – data loaders, embedding model, config, scripts
- Infrastructure/config:
- images (venv setup), pyproject.toml, GitHub Actions CI.

##### Key Architecture Insights from README
- Runs in a Python venv, optimized for Linux + NVIDIA GPU.
- Each subproject has its own README explaining how to launch via its `main.py` or `run.py`.
- Shared code supplies the training loop, config loading, logging, and checkpointing.

##### Main Components & Relationships
a. Ingestion & Configuration
- User/CLI & shell scripts (`run_local.sh`, `docker_run.sh`) →
- Virtualenv + dependencies (pyproject.toml, images/requirements.txt) →
- Config loader (core/config + project-specific YAML/JSON)
b. Data Pipeline
- reader/dataset + project/data modules →
- Dataset objects + Preprocessors →
- Batching (common/batch) →
- Device placement (common/device)
c. Model & Training
- Embedding modules (common/modules/embedding) + project/model code →
- Model + Loss (core/losses, core/loss_type) →
- Optimizer (optimizers/optimizer + project config) →
- Training loop (common/run_training or core/custom_training_loop) →
- Checkpointing (common/checkpointing/snapshot)
d. Evaluation & Metrics
- core/metrics + metrics/* modules →
- MetricMixin →
- Aggregation →
- Logging to Wandb or stdout (common/wandb, ml_logging)
e. Outputs
- Model checkpoints → Filesystem
- Logs & metrics → Wandb / console
- Artifacts/scripts for deployment

##### Architectural Patterns & Principles
- Layered architecture:
- Infrastructure & environment → Common library → Core training abstractions → Project-specific code
- Modular components: clear separation of data, model, optimizer, logging, metrics.
- Plugin/config-driven: new models or datasets can be added via config files without changing core logic.
- Dependency injection: components (optimizers, losses, metrics) supplied via configs.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Infrastructure"
        VENV["Init Venv Script"]:::infra
        REQ["Requirements"]:::infra
        PY["pyproject.toml"]:::infra
    end

    subgraph "Shared Libraries"
        subgraph common
            BATCH["Batching"]:::shared
            DEVICE["Device Management"]:::shared
            SNAP["Checkpointing"]:::shared
            FSUTIL["Filesystem Utils"]:::shared
            WADAPTER["Wandb Adapter"]:::shared
            EMBEMOD["Embedding Modules"]:::shared
            TRAINRUN["Training Runner"]:::shared
        end
        subgraph core
            CONFLOAD["Config Loader"]:::shared
            BASECONF["Base Config"]:::shared
            PIPELINE["Training Pipeline"]:::shared
            CTL["Custom Training Loop"]:::shared
            LOSSES["Loss Definitions"]:::shared
            METRICMIX["Metric Mixin & Defaults"]:::shared
        end
        subgraph metrics
            AUROC["AUROC"]:::shared
            AGG["Aggregation"]:::shared
            RCE["RCE"]:::shared
        end
        subgraph optimizers
            OPTIFACE["Optimizer Interface"]:::shared
            OPTICONF["Optimizer Config"]:::shared
        end
        subgraph reader
            DATASET["Dataset Abstractions"]:::shared
            DDS["DDS Connector"]:::shared
            RUTIL["Reader Utils"]:::shared
        end
        subgraph ml_logging
            ABSL["Absl Logging"]:::shared
            TORCHLOG["Torch Logging"]:::shared
        end
        subgraph tools
            PQ["Product Quantization"]:::shared
        end
    end

    subgraph "Project: home/recap"
        MAIN["Entrypoint main.py"]:::proj
        PCONF["Project Configs"]:::proj
        DDATA["Data Pipeline"]:::proj
        PMODEL["Model Definitions"]:::proj
        POPTI["Recap Optimizer"]:::proj
        SCRIPT["Local Run Script"]:::proj
    end

    subgraph "Project: twhin"
        RUN["Entrypoint run.py"]:::proj
        TCONFIG["Project Configs"]:::proj
        TDATA["Data Loaders"]:::proj
        TMODEL["Model Definitions"]:::proj
        TOPTI["Optimizer Wrapper"]:::proj
        TMETRICS["Metrics Hook"]:::proj
        DSCRIPTS["Docker Scripts"]:::proj
    end

    User["User/CLI"]:::external
    GPU["GPU (NVIDIA)"]:::external
    FS["Filesystem"]:::external
    WANDB["Wandb"]:::external

    User --> MAIN
    User --> RUN
    MAIN --> CONFLOAD
    RUN --> CONFLOAD
    CONFLOAD --> DATASET
    DATASET --> BATCH
    BATCH --> DEVICE
    DEVICE --> PMODEL
    DEVICE --> TMODEL
    PMODEL --> LOSSES
    TMODEL --> LOSSES
    LOSSES --> OPTIFACE
    OPTIFACE --> PIPELINE
    PIPELINE --> TRAINRUN
    TRAINRUN --> SNAP
    TRAINRUN --> METRICMIX
    TRAINRUN --> GPU
    METRICMIX --> AGG
    AGG --> WANDB
    SNAP --> FS

    click BATCH "https://github.com/twitter/the-algorithm-ml/blob/main/common/batch.py"
    click DEVICE "https://github.com/twitter/the-algorithm-ml/blob/main/common/device.py"
    click SNAP "https://github.com/twitter/the-algorithm-ml/blob/main/common/checkpointing/snapshot.py"
    click FSUTIL "https://github.com/twitter/the-algorithm-ml/blob/main/common/filesystem/util.py"
    click WADAPTER "https://github.com/twitter/the-algorithm-ml/blob/main/common/wandb.py"
    click EMBEMOD "https://github.com/twitter/the-algorithm-ml/blob/main/common/modules/embedding/embedding.py"
    click TRAINRUN "https://github.com/twitter/the-algorithm-ml/blob/main/common/run_training.py"
    click CONFLOAD "https://github.com/twitter/the-algorithm-ml/blob/main/core/config/config_load.py"
    click BASECONF "https://github.com/twitter/the-algorithm-ml/blob/main/core/config/base_config.py"
    click PIPELINE "https://github.com/twitter/the-algorithm-ml/blob/main/core/train_pipeline.py"
    click CTL "https://github.com/twitter/the-algorithm-ml/blob/main/core/custom_training_loop.py"
    click LOSSES "https://github.com/twitter/the-algorithm-ml/blob/main/core/losses.py"
    click METRICMIX "https://github.com/twitter/the-algorithm-ml/blob/main/core/metric_mixin.py"
    click AUROC "https://github.com/twitter/the-algorithm-ml/blob/main/metrics/auroc.py"
    click AGG "https://github.com/twitter/the-algorithm-ml/blob/main/metrics/aggregation.py"
    click RCE "https://github.com/twitter/the-algorithm-ml/blob/main/metrics/rce.py"
    click ABSL "https://github.com/twitter/the-algorithm-ml/blob/main/ml_logging/absl_logging.py"
    click TORCHLOG "https://github.com/twitter/the-algorithm-ml/blob/main/ml_logging/torch_logging.py"
    click OPTIFACE "https://github.com/twitter/the-algorithm-ml/blob/main/optimizers/optimizer.py"
    click OPTICONF "https://github.com/twitter/the-algorithm-ml/blob/main/optimizers/config.py"
    click DATASET "https://github.com/twitter/the-algorithm-ml/blob/main/reader/dataset.py"
    click DDS "https://github.com/twitter/the-algorithm-ml/blob/main/reader/dds.py"
    click RUTIL "https://github.com/twitter/the-algorithm-ml/blob/main/reader/utils.py"
    click PQ "https://github.com/twitter/the-algorithm-ml/blob/main/tools/pq.py"
    click VENV "https://github.com/twitter/the-algorithm-ml/blob/main/images/init_venv.sh"
    click REQ "https://github.com/twitter/the-algorithm-ml/blob/main/images/requirements.txt"
    click MAIN "https://github.com/twitter/the-algorithm-ml/blob/main/projects/home/recap/main.py"
    click PCONF "https://github.com/twitter/the-algorithm-ml/blob/main/projects/home/recap/config.py"
    click DDATA "https://github.com/twitter/the-algorithm-ml/blob/main/projects/home/recap/data/dataset.py"
    click PMODEL "https://github.com/twitter/the-algorithm-ml/blob/main/projects/home/recap/model/mask_net.py"
    click POPTI "https://github.com/twitter/the-algorithm-ml/blob/main/projects/home/recap/optimizer/optimizer.py"
    click SCRIPT "https://github.com/twitter/the-algorithm-ml/blob/main/projects/home/recap/script/run_local.sh"
    click RUN "https://github.com/twitter/the-algorithm-ml/blob/main/projects/twhin/run.py"
    click TCONFIG "https://github.com/twitter/the-algorithm-ml/blob/main/projects/twhin/config.py"
    click TDATA "https://github.com/twitter/the-algorithm-ml/blob/main/projects/twhin/data/data.py"
    click TMODEL "https://github.com/twitter/the-algorithm-ml/blob/main/projects/twhin/models/models.py"
    click TOPTI "https://github.com/twitter/the-algorithm-ml/blob/main/projects/twhin/optimizer.py"
    click TMETRICS "https://github.com/twitter/the-algorithm-ml/blob/main/projects/twhin/metrics.py"
    click DSCRIPTS "https://github.com/twitter/the-algorithm-ml/blob/main/projects/twhin/scripts/docker_run.sh"

    classDef shared fill:#ADD8E6,stroke:#000
    classDef proj fill:#90EE90,stroke:#000
    classDef external fill:#FFA500,stroke:#000
    classDef infra fill:#D3D3D3,stroke:#000
```
