How shenoynikhil/etflow works
Project Type & Purpose
• ET‑Flow is an open‑source scientific ML library (Python package) for generating molecular conformers via equivariant flow matching.
• It provides: data preprocessing, model training & evaluation pipelines, pretrained model loading, and programmatic APIs & CLI for inference.
Top‑Level Components
a. Configuration Layer
• configs/*.yaml
• env.yml, pyproject.toml
• Defines hyperparameters, dataset paths, model architecture choices.
b. Data Ingestion & Preprocessing
• scripts/prepare_data.py
• etflow/commons/io.py, featurization.py, xtb.py
• etflow/data/datamodule.py, dataset.py
• Input: raw GEOM data (RDKit folders)
• Output: processed datasets under $DATA_DIR/processed
c. Core Library (etflow package)
1. etflow/commons
• Shared utilities (I/O, featurization, covariance mat, sampling)
2. etflow/data
• PyTorch Lightning DataModule, Dataset abstractions
3. etflow/networks
• Equivariant transformers (torchmd_net) modules & dynamics
4. etflow/models
• High‑level Model interface (BaseFlow), loss definitions
5. etflow/schedulers
• Custom learning‐rate schedulers (CosineAnnealingWarmRestarts)
6. etflow/__init__.py & versioning
d. Training & Evaluation Pipelines
• scripts/train.py → instantiates DataModule, Model, Lightning Trainer → training loop
• scripts/eval.py / eval_xl.py → generation/inference
• scripts/eval_cov_mat.py → post‑hoc evaluation (RMSD metrics)
e. Interactive Notebooks & Tutorials
• tutorial.ipynb, generate_confs.ipynb
f. Packaging, CI/CD & Release
• pyproject.toml, setup for pip install etflow
• .github/workflows/release.yml for automated releases
• .pre-commit-config.yaml
g. External Dependencies & Resources
• PyTorch, PyTorch Lightning, torchmd-net, e3-diffusion-for-molecules, RDKit
• GEOM-DRUGS & QM9 datasets (external storage)
• Zenodo for checkpoint hosting
Relationships & Data Flow
• Config Files feed into both preprocessing & training scripts.
• prepare_data.py reads raw data → applies etflow.commons featurization → writes processed data.
• train.py loads configs → DataModule loads processed data → Model (with networks + schedulers) → PyTorch Lightning Trainer → checkpoints & logs
• eval.py/xl.py load trained checkpoint & config → generate conformers → write generated_files.pkl
• eval_cov_mat.py reads generated files → computes metrics.
• BaseFlow.from_default API uses pre‑packed configs & checkpoints (downloaded via Zenodo) for inference.
Architectural Patterns & Principles
• Modular design: clear separation between Commons, Data, Networks, Models, Schedulers
• Dependency injection via YAML config: decouples hyperparameters from code
• PyTorch Lightning: standardized Trainer/DataModule/Callback pattern
• Extensibility points: additional networks, schedulers, featurizers can be plugged in
Essential Elements Checklist
• All top‑level folders & their primary roles
• Configurations driving behavior
• Data preprocessing pipeline
• Model composition (networks + loss + schedulers)
• Training vs. evaluation pathways
• External integrations (datasets, checkpoints, CI/CD)