# mims-harvard/tdc architecture

> Architecture diagram of the GitHub repository mims-harvard/tdc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mims-harvard/tdc
- Repository: https://github.com/mims-harvard/tdc

## Overview

##### Project Purpose and Type
- PyTDC is an open-source machine-learning platform built for therapeutics, integrating heterogeneous, continuously updated biomedical data with state‐of‐the‐art models for training, evaluation, and inference.
- Its API-first design, microservice architecture, and modular “Problem – Learning Task – Dataset” structure are important; it handles single-instance prediction, multi-instance prediction, and generation tasks.

##### Main Components
- Data Sources & Ingestion:
– External data sources (e.g., Harvard Dataverse, continuously updated datasets).
– Built-in dataset configuration modules (folder: tdc/dataset_configs) that manage dataset information.
- Core API & Data Loaders:
– Hierarchical “Problem → Learning Task → Dataset” structure.
– Data loader APIs and functions (e.g., in tdc/single_pred, tdc/multi_pred, tdc/generation) that provide structured access to the datasets.
- Data Functions & Utilities:
– Chemical utilities (tdc/chem_utils), feature generators (tdc/feature_generators), and miscellaneous utilities (tdc/utils) that process and transform data.
- Model Server & Inference:
– Model server components (tdc/model_server) including model loaders, tokenizers, and inference endpoints.
- Evaluation & Benchmarking:
– Evaluation modules and benchmark groups (both in the tdc subdirectories and examples) that cover tasks like data splitting, performance metrics, and leaderboards.
- Oracles for Molecule Generation:
– Oracles and generative utilities (see modules within tdc/generation and examples/generation) for goal-directed and distribution-learning tasks.
- CI/CD, Documentation, and Testing:
– Build and test configurations (.circleci, .github workflows, pre-commit, RTD configuration) indicating automated testing, code quality, and continuous documentation.
– Extensive documentation (docs folder) and tutorials (tutorials directory) to showcase usage and workflows.

##### Relationships and Interactions Between Components
- Data Flow:
– External data sources feed into dataset configuration modules that “ingest” data into the system.
– Data loaders (via the API) then deliver this organized data to the specific learning tasks (single_pred, multi_pred, generation).
- Processing Flow:
– Utility components (chem_utils, feature_generators, data functions, etc.) process the raw data and support tasks like splitting and featurization.
– The processed data is then passed on to models (via the model server, tokenizers, and loaders), where training/inference happens.
- Evaluation Flow:
– Once models are run, evaluation modules (evaluators, benchmark groups, oracles) capture performance metrics.
– These are used to update leaderboards and benchmarks.
- Deployment & Continuous Integration:
– CI/CD pipelines execute tests, quality checks, and documentation builds, ensuring reliable and reproducible workflows.
– The system server (model_server) also exposes inference endpoints for production use.

##### Architectural Patterns and Design Principles
- API-First Design: Emphasizes that every dataset and learning task is available via high-level API calls.
- Modular/MVC Patterns: There’s a clear separation between the “models/view” (data loaders, oracles) and the underlying “controller” functions in utilities and evaluators.
- Microservice-Like Structure: Components like the model server interact with independently managed datasets and external repositories.
- Hierarchical Organization: Three-tier design – problems, learning tasks, and datasets – provides clarity in user workflows and extensions.

##### Technologies and Frameworks
- Python and its ecosystem (numpy, pandas, scikit-learn, etc.) serve as the primary technology.
- CI/CD tooling by CircleCI and GitHub Actions for orchestration.
- Documentation tools like ReadTheDocs and Sphinx (docs/conf.py).
- Notebook examples and demo scripts (Jupyter notebooks in tutorials and examples) to illustrate the system usage.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Nodes with Classes
    eds["External Data Sources"]:::external
    
    subgraph "Data Layer"
        ds["Dataset Configs"]:::data
    end

    subgraph "API Layer"
        sp["Single-instance Loader"]:::api
        mp["Multi-instance Loader"]:::api
        gt["Generation Tasks"]:::api
    end

    subgraph "Utilities Layer"
        cu["Chemical Utilities"]:::utilities
        fg["Feature Generators"]:::utilities
        mu["Miscellaneous Utilities"]:::utilities
    end

    subgraph "Inference Layer"
        ms["Model Server"]:::model
        eb["Benchmark Groups"]:::evaluation
    end

    subgraph "Supporting Infrastructure"
        ci["CI/CD Pipelines"]:::support
        doc["Documentation"]:::support
        te["Tutorials & Examples"]:::support
    end

    %% Connections - Data Flow
    eds --> ds
    ds --> sp
    ds --> mp
    ds --> gt

    sp --> cu
    mp --> fg
    gt --> mu

    cu --> ms
    fg --> ms
    mu --> ms

    ms --> eb
    eb --> te

    %% Supporting Infrastructure Connections
    ci --> ds
    ci --> ms
    ci --> doc
    doc --> te

    %% Click Events
    click ds "https://github.com/mims-harvard/tdc/tree/main/tdc/dataset_configs"
    click sp "https://github.com/mims-harvard/tdc/tree/main/tdc/single_pred"
    click mp "https://github.com/mims-harvard/tdc/tree/main/tdc/multi_pred"
    click gt "https://github.com/mims-harvard/tdc/tree/main/tdc/generation"
    click cu "https://github.com/mims-harvard/tdc/tree/main/tdc/chem_utils"
    click fg "https://github.com/mims-harvard/tdc/tree/main/tdc/feature_generators"
    click mu "https://github.com/mims-harvard/tdc/tree/main/tdc/utils"
    click ms "https://github.com/mims-harvard/tdc/tree/main/tdc/model_server"
    click eb "https://github.com/mims-harvard/tdc/tree/main/tdc/benchmark_group"
    click ci "https://github.com/mims-harvard/tdc/blob/main/.circleci"
    click ci "https://github.com/mims-harvard/tdc/tree/main/.github/workflows"
    click doc "https://github.com/mims-harvard/tdc/tree/main/docs"
    click te "https://github.com/mims-harvard/tdc/tree/main/tutorials"
    click te "https://github.com/mims-harvard/tdc/tree/main/examples"

    %% Class Styles
    classDef external fill:#B2DFDB,stroke:#004D40,stroke-width:2px;
    classDef data fill:#FFE0B2,stroke:#E65100,stroke-width:2px;
    classDef api fill:#BBDEFB,stroke:#1E88E5,stroke-width:2px;
    classDef utilities fill:#C8E6C9,stroke:#2E7D32,stroke-width:2px;
    classDef model fill:#D1C4E9,stroke:#512DA8,stroke-width:2px;
    classDef evaluation fill:#FFCDD2,stroke:#C62828,stroke-width:2px;
    classDef support fill:#F0F4C3,stroke:#827717,stroke-width:2px;
```
