How atomscience-org/ampl works
Project type and purpose
• AMPL is an open-source, end-to-end Python software pipeline for molecular data curation, featurization, model building, prediction, and analysis in drug discovery. It is packaged as a library (namespace package atomsci.ddm), a command-line tool, and a collection of Jupyter notebooks.
High-level architectural overview
• This is not a traditional frontend/backend web application, but rather a modular, configuration-driven library and CLI tool. The core is a “pipeline orchestrator” that reads JSON config files, invokes a series of modules (data curation, featurization, splitting, training, prediction, visualization), and writes outputs (models, metrics, plots) to disk or external services.
Main components
a. User interfaces
• CLI entrypoint (model_pipeline.py, hyperparam_search_wrapper.py)
• Python API (importable modules under atomsci.ddm.pipeline, atomsci.ddm.utils)
• Jupyter notebooks (atomsci/ddm/examples/tutorials)
b. Configuration Manager
• JSON parameter files (under atomsci/ddm/config*, tests/integrative/.../jsons)
• Parameter parser (atomsci/ddm/pipeline/parameter_parser.py)
c. Pipeline Orchestrator
• model_pipeline.py / model_pipeline function
• Reads config → sets up sequence of tasks → error handling / logging
d. Core modules (each invoked by orchestrator)
1. Data Curation (atomsci.ddm.utils.curate_data, data_curation_functions)
2. Featurization (atomsci.ddm.pipeline.featurization)
3. Splitting (atomsci.ddm.pipeline.splitting, MultitaskScaffoldSplit, temporal_splitter)
4. Model Wrapper & Training (model_wrapper.py, model_pipeline.py)
5. Hyperparameter Search (hyperparam_search_wrapper.py)
6. Prediction (predict_from_model.py)
7. Visualization & Analysis (perf_plots.py, diversity_plots.py, split_response_dist_plots.py)
e. Storage and persistence
• Local file system: datasets, intermediate descriptor files, trained model archives (.tar.gz)
• Model registry / tracker (optional external service via atomsci.clients)
• Versioning and checksums (checksum_utils.py, model_version_utils.py)
f. External dependencies and services
• Chemistry toolkits: RDKit, Mordred, MOE (optional)
• ML frameworks: DeepChem, TensorFlow, PyTorch, DGL, scikit-learn, XGBoost, HyperOpt
• Containerization: Docker images for CPU, GPU, ARM
• CI/CD: GitHub Actions (build, lint, pytest), ReadTheDocs for docs
• HPC/SLURM for long-running tests (via pytest_slurm.sh)
g. Development & CI
• Namespace packaging (setup.py, build.sh)
• Virtual environments and pip requirement sets (under pip/)
• GitHub workflows (.github/workflows/*.yml)
Component interactions and data flow
1. User invokes CLI or imports pipeline from Python/Jupyter.
2. Configuration Manager loads JSON parameters.
3. Pipeline Orchestrator schedules modules in sequence:
– Data Curation reads raw source data, applies cleaning.
– Featurization derives molecular descriptors/fingerprints.
– Splitting partitions data (random, scaffold, temporal).
– Model Training builds and evaluates model(s) (RF, XGB, NN, GC).
– Hyperparameter Search (if enabled) iterates model configurations.
– Prediction applies trained model to external holdout data.
– Visualization generates plots and performance summaries.
4. Each module calls external libraries and writes outputs to disk or optional remote tracker.
5. Tests exercise all components, downloading datasets if needed (unit and integrative tests).
Architectural patterns and principles
• Pipeline (chain-of-responsibility): discrete stages connected by the orchestrator
• Config-driven execution: behavior defined by JSON parameters
• Modular separation of concerns: curation, featurization, splitting, modeling, visualization
• Namespace packaging for easy development (-e .)
• Containerization for reproducible environment (Docker CPU/GPU/ARM images)
• CI/CD integration to enforce quality (lint, test, docs)