How twitter/the-algorithm-ml works
Project Type & Purpose
• This is an open-source ML toolkit for training and evaluating two embedding/ranking models.
• It provides shared libraries (“common”, “core”, “metrics”, “ml_logging”, “optimizers”, “reader”), plus two end-to-end projects under projects/home/recap and projects/twhin.
File Structure
• Top-level libraries:
• common (batching, device management, checkpointing, filesystem utils, embedding modules)
• core (configuration, training loops, pipeline, losses, metrics)
• metrics (AUROC, aggregation, RCE)
• ml_logging (absl and torch logging adapters)
• optimizers (configurable optimizer interface)
• reader (data ingestion & utils)
• tools (utilities like PQ)
• Projects:
• home/recap – data generation & preprocessing, model definitions (mask_net, mlp, numeric_calibration), entrypoint, scripts
• twhin – data loaders, embedding model, config, scripts
• Infrastructure/config:
• images (venv setup), pyproject.toml, GitHub Actions CI.
Key Architecture Insights from README
• Runs in a Python venv, optimized for Linux + NVIDIA GPU.
• Each subproject has its own README explaining how to launch via its main.py or run.py.
• Shared code supplies the training loop, config loading, logging, and checkpointing.
Main Components & Relationships
a. Ingestion & Configuration
• User/CLI & shell scripts (run_local.sh, docker_run.sh) →
• Virtualenv + dependencies (pyproject.toml, images/requirements.txt) →
• Config loader (core/config + project-specific YAML/JSON)
b. Data Pipeline
• reader/dataset + project/data modules →
• Dataset objects + Preprocessors →
• Batching (common/batch) →
• Device placement (common/device)
c. Model & Training
• Embedding modules (common/modules/embedding) + project/model code →
• Model + Loss (core/losses, core/loss_type) →
• Optimizer (optimizers/optimizer + project config) →
• Training loop (common/run_training or core/custom_training_loop) →
• Checkpointing (common/checkpointing/snapshot)
d. Evaluation & Metrics
• core/metrics + metrics/* modules →
• MetricMixin →
• Aggregation →
• Logging to Wandb or stdout (common/wandb, ml_logging)
e. Outputs
• Model checkpoints → Filesystem
• Logs & metrics → Wandb / console
• Artifacts/scripts for deployment
Architectural Patterns & Principles
• Layered architecture:
• Infrastructure & environment → Common library → Core training abstractions → Project-specific code
• Modular components: clear separation of data, model, optimizer, logging, metrics.
• Plugin/config-driven: new models or datasets can be added via config files without changing core logic.
• Dependency injection: components (optimizers, losses, metrics) supplied via configs.