# atomscience-org/ampl architecture

> Architecture diagram of the GitHub repository atomscience-org/ampl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/atomscience-org/ampl
- Repository: https://github.com/atomscience-org/ampl

## Overview

##### Project type and purpose
- AMPL is an open-source, end-to-end Python software pipeline for molecular data curation, featurization, model building, prediction, and analysis in drug discovery.  It is packaged as a library (namespace package `atomsci.ddm`), a command-line tool, and a collection of Jupyter notebooks.

##### High-level architectural overview
- This is not a traditional frontend/backend web application, but rather a modular, configuration-driven library and CLI tool.  The core is a “pipeline orchestrator” that reads JSON config files, invokes a series of modules (data curation, featurization, splitting, training, prediction, visualization), and writes outputs (models, metrics, plots) to disk or external services.

##### Main components
a. User interfaces
- CLI entrypoint (`model_pipeline.py`, `hyperparam_search_wrapper.py`)
- Python API (importable modules under `atomsci.ddm.pipeline`, `atomsci.ddm.utils`)
- Jupyter notebooks (`atomsci/ddm/examples/tutorials`)
b. Configuration Manager
- JSON parameter files (under `atomsci/ddm/config*`, `tests/integrative/.../jsons`)
- Parameter parser (`atomsci/ddm/pipeline/parameter_parser.py`)
c. Pipeline Orchestrator
- `model_pipeline.py` / `model_pipeline` function
- Reads config → sets up sequence of tasks → error handling / logging
d. Core modules (each invoked by orchestrator)
1. Data Curation (`atomsci.ddm.utils.curate_data`, `data_curation_functions`)
2. Featurization (`atomsci.ddm.pipeline.featurization`)
3. Splitting (`atomsci.ddm.pipeline.splitting`, `MultitaskScaffoldSplit`, `temporal_splitter`)
4. Model Wrapper & Training (`model_wrapper.py`, `model_pipeline.py`)
5. Hyperparameter Search (`hyperparam_search_wrapper.py`)
6. Prediction (`predict_from_model.py`)
7. Visualization & Analysis (`perf_plots.py`, `diversity_plots.py`, `split_response_dist_plots.py`)
e. Storage and persistence
- Local file system: datasets, intermediate descriptor files, trained model archives (.tar.gz)
- Model registry / tracker (optional external service via `atomsci.clients`)
- Versioning and checksums (`checksum_utils.py`, `model_version_utils.py`)
f. External dependencies and services
- Chemistry toolkits: RDKit, Mordred, MOE (optional)
- ML frameworks: DeepChem, TensorFlow, PyTorch, DGL, scikit-learn, XGBoost, HyperOpt
- Containerization: Docker images for CPU, GPU, ARM
- CI/CD: GitHub Actions (build, lint, pytest), ReadTheDocs for docs
- HPC/SLURM for long-running tests (via `pytest_slurm.sh`)
g. Development & CI
- Namespace packaging (`setup.py`, `build.sh`)
- Virtual environments and pip requirement sets (under `pip/`)
- GitHub workflows (`.github/workflows/*.yml`)

##### Component interactions and data flow
1. User invokes CLI or imports pipeline from Python/Jupyter.
2. Configuration Manager loads JSON parameters.
3. Pipeline Orchestrator schedules modules in sequence:
– Data Curation reads raw source data, applies cleaning.
– Featurization derives molecular descriptors/fingerprints.
– Splitting partitions data (random, scaffold, temporal).
– Model Training builds and evaluates model(s) (RF, XGB, NN, GC).
– Hyperparameter Search (if enabled) iterates model configurations.
– Prediction applies trained model to external holdout data.
– Visualization generates plots and performance summaries.
4. Each module calls external libraries and writes outputs to disk or optional remote tracker.
5. Tests exercise all components, downloading datasets if needed (unit and integrative tests).

##### Architectural patterns and principles
- Pipeline (chain-of-responsibility): discrete stages connected by the orchestrator
- Config-driven execution: behavior defined by JSON parameters
- Modular separation of concerns: curation, featurization, splitting, modeling, visualization
- Namespace packaging for easy development (`-e .`)
- Containerization for reproducible environment (Docker CPU/GPU/ARM images)
- CI/CD integration to enforce quality (lint, test, docs)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Docker/Environment" 
        direction TB
        %% Entry Points
        subgraph "User Interfaces" 
            direction TB
            CLI["CLI Entrypoint\nmodel_pipeline.py"]:::entry
            HPO_CLI["CLI Entrypoint\nhyperparam_search_wrapper.py"]:::entry
            API["Python API\natomsci.ddm.pipeline/\n& utils"]:::entry
            NB_TUTORIALS["Jupyter Notebooks\nTutorials"]:::entry
            NB_DELANEY["Jupyter Notebook\nDelaney Demo"]:::entry
        end

        CLI -->|invokes| CONFIG
        HPO_CLI -->|invokes| CONFIG
        API -->|imports| CONFIG
        NB_TUTORIALS -->|uses| API
        NB_DELANEY -->|uses| API

        %% Configuration Manager
        CONFIG["Configuration Manager\n(JSON files + Parser)"]:::config
        click CONFIG "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/config/*.json"
        click CONFIG "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/config_files/*.json"
        click CONFIG "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/test/integrative/**/jsons/*.json"
        click CONFIG "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/pipeline/parameter_parser.py"

        %% Orchestrator
        ORCH["Pipeline Orchestrator\nmodel_pipeline"]:::core
        click ORCH "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/pipeline/model_pipeline.py"

        CONFIG -->|loads params| ORCH

        %% Pipeline Modules
        subgraph "Core Pipeline Modules" 
            direction TB
            DC["Data Curation"]:::core
            FEAT["Featurization"]:::core
            SPLIT["Splitting"]:::core
            TRAIN["Model Training"]:::core
            HPO["Hyperparameter Search"]:::core
            PRED["Prediction"]:::core
            VIZ["Visualization & Analysis"]:::core

            ORCH -->|runs| DC
            DC -->|SMILES→descriptors| FEAT
            FEAT -->|descriptors→data| SPLIT
            SPLIT -->|train/test sets| TRAIN
            TRAIN -->|trained model| HPO
            HPO -->|best model| PRED
            PRED -->|predictions| VIZ

            click DC "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/utils/curate_data.py"
            click DC "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/utils/data_curation_functions.py"
            click FEAT "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/pipeline/featurization.py"
            click SPLIT "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/pipeline/splitting.py"
            click SPLIT "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/pipeline/MultitaskScaffoldSplit.py"
            click SPLIT "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/pipeline/temporal_splitter.py"
            click TRAIN "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/pipeline/model_wrapper.py"
            click HPO "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/utils/hyperparam_search_wrapper.py"
            click PRED "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/pipeline/predict_from_model.py"
            click VIZ "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/pipeline/perf_plots.py"
            click VIZ "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/pipeline/diversity_plots.py"
            click VIZ "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/utils/split_response_dist_plots.py"
        end

        %% Storage & Persistence
        subgraph "Storage & Persistence" 
            direction TB
            DATASTORE["Data Store"]:::storage
            MODELREG["Model Registry"]:::storage

            VIZ -->|outputs| DATASTORE
            TRAIN -->|model .tar.gz| MODELREG

            click DATASTORE "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/utils/datastore_functions.py"
            click MODELREG "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/pipeline/model_tracker.py"
            click MODELREG "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/utils/model_version_utils.py"
            click MODELREG "https://github.com/atomscience-org/ampl/blob/master/atomsci/ddm/utils/checksum_utils.py"
        end

        %% External Dependencies
        subgraph "External Libraries/Services" 
            direction TB
            EXLIB["Chemistry Toolkits & ML Frameworks"]:::external
            EXLIB --> DC
            EXLIB --> FEAT
            EXLIB --> TRAIN
        end

        %% CI/CD & Containerization
        subgraph "CI/CD & Containerization" 
            direction TB
            CI["GitHub Actions & Tests"]:::ci
            DOCKER["Docker Images"]:::ci
            CI --> ORCH
            DOCKER --> ORCH

            click CI "https://github.com/atomscience-org/ampl/blob/master/.github/workflows/*.yml"
            click DOCKER "https://github.com/atomscience-org/ampl/blob/master/Dockerfile.cpu"
            click DOCKER "https://github.com/atomscience-org/ampl/blob/master/Dockerfile.gpu"
            click DOCKER "https://github.com/atomscience-org/ampl/blob/master/Dockerfile.arm"
        end
    end

    classDef entry fill:#9f6,stroke:#333,stroke-width:1px;
    classDef config fill:#ff9,stroke:#333,stroke-width:1px;
    classDef core fill:#6cf,stroke:#333,stroke-width:1px;
    classDef storage fill:#fe9,stroke:#333,stroke-width:1px;
    classDef external fill:#f96,stroke:#333,stroke-width:1px;
    classDef ci fill:#ccc,stroke:#333,stroke-width:1px;
```
