# shenoynikhil/etflow architecture

> Architecture diagram of the GitHub repository shenoynikhil/etflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shenoynikhil/etflow
- Repository: https://github.com/shenoynikhil/etflow

## Overview

##### Project Type & Purpose
- ET‑Flow is an open‑source scientific ML library (Python package) for generating molecular conformers via equivariant flow matching.
- It provides: data preprocessing, model training & evaluation pipelines, pretrained model loading, and programmatic APIs & CLI for inference.

##### Top‑Level Components
a. **Configuration Layer**
- configs/*.yaml
- env.yml, pyproject.toml
- Defines hyperparameters, dataset paths, model architecture choices.

b. **Data Ingestion & Preprocessing**
- scripts/prepare_data.py
- etflow/commons/io.py, featurization.py, xtb.py
- etflow/data/datamodule.py, dataset.py
- Input: raw GEOM data (RDKit folders)
- Output: processed datasets under `$DATA_DIR/processed`

c. **Core Library (etflow package)**
1. etflow/commons
- Shared utilities (I/O, featurization, covariance mat, sampling)
2. etflow/data
- PyTorch Lightning DataModule, Dataset abstractions
3. etflow/networks
- Equivariant transformers (torchmd_net) modules & dynamics
4. etflow/models
- High‑level Model interface (BaseFlow), loss definitions
5. etflow/schedulers
- Custom learning‐rate schedulers (CosineAnnealingWarmRestarts)
6. etflow/__init__.py & versioning

d. **Training & Evaluation Pipelines**
- scripts/train.py → instantiates DataModule, Model, Lightning Trainer → training loop
- scripts/eval.py / eval_xl.py → generation/inference
- scripts/eval_cov_mat.py → post‑hoc evaluation (RMSD metrics)

e. **Interactive Notebooks & Tutorials**
- tutorial.ipynb, generate_confs.ipynb

f. **Packaging, CI/CD & Release**
- pyproject.toml, setup for `pip install etflow`
- .github/workflows/release.yml for automated releases
- .pre-commit-config.yaml

g. **External Dependencies & Resources**
- PyTorch, PyTorch Lightning, torchmd-net, e3-diffusion-for-molecules, RDKit
- GEOM-DRUGS & QM9 datasets (external storage)
- Zenodo for checkpoint hosting

##### Relationships & Data Flow
- **Config Files** feed into both preprocessing & training scripts.
- **prepare_data.py** reads raw data → applies etflow.commons featurization → writes processed data.
- **train.py** loads configs → DataModule loads processed data → Model (with networks + schedulers) → PyTorch Lightning Trainer → checkpoints & logs
- **eval.py/xl.py** load trained checkpoint & config → generate conformers → write `generated_files.pkl`
- **eval_cov_mat.py** reads generated files → computes metrics.
- **BaseFlow.from_default** API uses pre‑packed configs & checkpoints (downloaded via Zenodo) for inference.

##### Architectural Patterns & Principles
- **Modular design**: clear separation between Commons, Data, Networks, Models, Schedulers
- **Dependency injection via YAML config**: decouples hyperparameters from code
- **PyTorch Lightning**: standardized Trainer/DataModule/Callback pattern
- **Extensibility points**: additional networks, schedulers, featurizers can be plugged in

##### Essential Elements Checklist
- All top‑level folders & their primary roles
- Configurations driving behavior
- Data preprocessing pipeline
- Model composition (networks + loss + schedulers)
- Training vs. evaluation pathways
- External integrations (datasets, checkpoints, CI/CD)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration Layer
    subgraph "Configuration Layer"
        direction TB
        CFG_YAML["configs/*.yaml"]:::config
        ENV["env.yml"]:::config
        PYPJ["pyproject.toml"]:::config
    end

    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        direction TB
        RAW["Raw GEOM Data"]:::data_store
        PREP["prepare_data.py"]:::preproc
        IO_UTILS["etflow/commons/io.py"]:::preproc
        FEAT["etflow/commons/featurization.py"]:::preproc
        XTB["etflow/commons/xtb.py"]:::preproc
        DMOD["etflow/data/datamodule.py"]:::preproc
        DATASET["etflow/data/dataset.py"]:::preproc
        PROCD["Processed Data"]:::data_store
    end

    %% Core Library
    subgraph "Core Library (etflow package)"
        direction TB
        subgraph Commons
            direction TB
            C_CFG["etflow/commons/configs.py"]:::core
            C_IO["etflow/commons/io.py"]:::core
            C_FEAT["etflow/commons/featurization.py"]:::core
            C_XTB["etflow/commons/xtb.py"]:::core
            C_COV["etflow/commons/covmat.py"]:::core
            C_SAMP["etflow/commons/sample.py"]:::core
            C_UTIL["etflow/commons/utils.py"]:::core
        end
        subgraph DataModules
            direction TB
            D_INIT["etflow/data/__init__.py"]:::core
            D_DM["etflow/data/datamodule.py"]:::core
            D_DS["etflow/data/dataset.py"]:::core
        end
        subgraph Networks
            direction TB
            N_INIT["etflow/networks/__init__.py"]:::core
            N_TM_INIT["etflow/networks/torchmd_net/__init__.py"]:::core
            N_DYN["etflow/networks/torchmd_net/model_dynamics.py"]:::core
            N_MODS["etflow/networks/torchmd_net/modules.py"]:::core
            N_UTILS["etflow/networks/torchmd_net/utils.py"]:::core
        end
        subgraph Models
            direction TB
            M_INIT["etflow/models/__init__.py"]:::core
            M_BASE["etflow/models/base.py"]:::core
            M_LOSS["etflow/models/loss.py"]:::core
            M_MODEL["etflow/models/model.py"]:::core
            M_UTIL["etflow/models/utils.py"]:::core
        end
        subgraph Schedulers
            direction TB
            SCHED_INIT["etflow/schedulers/__init__.py"]:::core
            S_COS["etflow/schedulers/CosineAnnealingWarmRestarts.py"]:::core
        end
        subgraph Metadata
            direction TB
            PKG_INIT["etflow/__init__.py"]:::core
            VERS["etflow/_version.py"]:::core
        end
    end

    %% Training & Evaluation Pipelines
    subgraph "Training & Evaluation Pipelines"
        direction TB
        TRAIN["scripts/train.py"]:::script
        EVAL["scripts/eval.py"]:::script
        EVAL_XL["scripts/eval_xl.py"]:::script
        EVAL_COV["scripts/eval_cov_mat.py"]:::script
        EVAL_PROP["scripts/eval_prop.py"]:::script
        S_UTIL["scripts/utils.py"]:::script
    end

    %% Notebooks
    subgraph "Interactive Notebooks & Tutorials"
        direction TB
        NB1["tutorial.ipynb"]:::notebook
        NB2["generate_confs.ipynb"]:::notebook
    end

    %% Packaging & CI/CD
    subgraph "Packaging, CI/CD & Release"
        direction TB
        PPY["pyproject.toml"]:::pack
        CI[".github/workflows/release.yml"]:::pack
        PRECOM[".pre-commit-config.yaml"]:::pack
        GITIGN[".gitignore"]:::pack
        LIC["LICENSE"]:::pack
        README["README.md"]:::pack
    end

    %% External Dependencies & Resources
    subgraph "External & Static Resources"
        direction TB
        TORCH["PyTorch / Lightning / torchmd-net"]:::external
        RDKit["RDKit"]:::external
        DATASET_EXT["GEOM-DRUGS & QM9"]:::external
        ZENODO["Zenodo (checkpoints)"]:::external
        CI_CD["GitHub Actions"]:::external
    end

    %% Data Flows
    CFG_YAML -.->|config| PREP
    ENV -.->|env| PREP
    PREP -->|featurize| IO_UTILS & FEAT & XTB
    IO_UTILS & FEAT & XTB -->|generate| PROCD
    RAW -->|input| PREP
    PROCD -->|load| DMOD
    DMOD -->|provide| TRAIN
    CFG_YAML -.->|config| TRAIN
    ENV -.->|env| TRAIN
    TRAIN -->|uses| D_DM & N_DYN & S_COS
    N_DYN -->|calls| N_MODS & N_UTILS
    TRAIN -->|logs & ckpt| CKPT["Checkpoints"]:::data_store
    TRAIN -->|calls| VERS
    TRAIN -->|calls| PKG_INIT
    TRAIN -->|trains| M_BASE & M_MODEL & M_LOSS
    TRAIN -->|schedule| S_COS
    TRAIN -->|lightning| TORCH
    EVAL -->|load ckpt| CKPT
    EVAL -->|generate confs| GEN["generated_files.pkl"]:::data_store
    EVAL_XL -->|extended gen| GEN
    EVAL_COV -->|compute metrics| GEN & PROCD
    EVAL_PROP -->|compute props| GEN
    GEN -->|input| EVAL_COV & EVAL_PROP

    %% Click Events
    click CFG_YAML "https://github.com/shenoynikhil/etflow/blob/main/configs/drugs-base.yaml"
    click ENV "https://github.com/shenoynikhil/etflow/blob/main/env.yml"
    click PYPJ "https://github.com/shenoynikhil/etflow/blob/main/pyproject.toml"
    click PREP "https://github.com/shenoynikhil/etflow/blob/main/scripts/prepare_data.py"
    click IO_UTILS "https://github.com/shenoynikhil/etflow/blob/main/etflow/commons/io.py"
    click FEAT "https://github.com/shenoynikhil/etflow/blob/main/etflow/commons/featurization.py"
    click XTB "https://github.com/shenoynikhil/etflow/blob/main/etflow/commons/xtb.py"
    click DMOD "https://github.com/shenoynikhil/etflow/blob/main/etflow/data/datamodule.py"
    click DATASET "https://github.com/shenoynikhil/etflow/blob/main/etflow/data/dataset.py"
    click C_CFG "https://github.com/shenoynikhil/etflow/blob/main/etflow/commons/configs.py"
    click C_IO "https://github.com/shenoynikhil/etflow/blob/main/etflow/commons/io.py"
    click C_FEAT "https://github.com/shenoynikhil/etflow/blob/main/etflow/commons/featurization.py"
    click C_XTB "https://github.com/shenoynikhil/etflow/blob/main/etflow/commons/xtb.py"
    click C_COV "https://github.com/shenoynikhil/etflow/blob/main/etflow/commons/covmat.py"
    click C_SAMP "https://github.com/shenoynikhil/etflow/blob/main/etflow/commons/sample.py"
    click C_UTIL "https://github.com/shenoynikhil/etflow/blob/main/etflow/commons/utils.py"
    click D_DM "https://github.com/shenoynikhil/etflow/blob/main/etflow/data/datamodule.py"
    click D_DS "https://github.com/shenoynikhil/etflow/blob/main/etflow/data/dataset.py"
    click N_DYN "https://github.com/shenoynikhil/etflow/blob/main/etflow/networks/torchmd_net/model_dynamics.py"
    click N_MODS "https://github.com/shenoynikhil/etflow/blob/main/etflow/networks/torchmd_net/modules.py"
    click N_UTILS "https://github.com/shenoynikhil/etflow/blob/main/etflow/networks/torchmd_net/utils.py"
    click M_BASE "https://github.com/shenoynikhil/etflow/blob/main/etflow/models/base.py"
    click M_LOSS "https://github.com/shenoynikhil/etflow/blob/main/etflow/models/loss.py"
    click M_MODEL "https://github.com/shenoynikhil/etflow/blob/main/etflow/models/model.py"
    click M_UTIL "https://github.com/shenoynikhil/etflow/blob/main/etflow/models/utils.py"
    click S_COS "https://github.com/shenoynikhil/etflow/blob/main/etflow/schedulers/CosineAnnealingWarmRestarts.py"
    click PKG_INIT "https://github.com/shenoynikhil/etflow/blob/main/etflow/__init__.py"
    click VERS "https://github.com/shenoynikhil/etflow/blob/main/etflow/_version.py"
    click TRAIN "https://github.com/shenoynikhil/etflow/blob/main/scripts/train.py"
    click EVAL "https://github.com/shenoynikhil/etflow/blob/main/scripts/eval.py"
    click EVAL_XL "https://github.com/shenoynikhil/etflow/blob/main/scripts/eval_xl.py"
    click EVAL_COV "https://github.com/shenoynikhil/etflow/blob/main/scripts/eval_cov_mat.py"
    click EVAL_PROP "https://github.com/shenoynikhil/etflow/blob/main/scripts/eval_prop.py"
    click S_UTIL "https://github.com/shenoynikhil/etflow/blob/main/scripts/utils.py"
    click NB1 "https://github.com/shenoynikhil/etflow/blob/main/tutorial.ipynb"
    click NB2 "https://github.com/shenoynikhil/etflow/blob/main/generate_confs.ipynb"
    click PPY "https://github.com/shenoynikhil/etflow/blob/main/pyproject.toml"
    click CI "https://github.com/shenoynikhil/etflow/blob/main/.github/workflows/release.yml"
    click PRECOM "https://github.com/shenoynikhil/etflow/blob/main/.pre-commit-config.yaml"
    click GITIGN "https://github.com/shenoynikhil/etflow/blob/main/.gitignore"
    click LIC "https://github.com/shenoynikhil/etflow/tree/main/LICENSE"
    click README "https://github.com/shenoynikhil/etflow/blob/main/README.md"

    %% Styles
    classDef config fill:#FFE4B3,stroke:#CC9900
    classDef preproc fill:#FFD9B3,stroke:#E68A00
    classDef core fill:#CCE5FF,stroke:#3385FF
    classDef script fill:#D5FFD6,stroke:#33CC33
    classDef notebook fill:#F2E5FF,stroke:#9933CC
    classDef pack fill:#F2F2F2,stroke:#666666
    classDef data_store fill:#E6E6E6,stroke:#999999,stroke-dasharray: 5 2
    classDef external fill:#F2F2F2,stroke:#999999,stroke-dasharray: 2 2
    class CFG_YAML,ENV,PYPJ config
    class RAW,PROCD,CKPT,GEN data_store
```
