# datamol-io/graphium architecture

> Architecture diagram of the GitHub repository datamol-io/graphium, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datamol-io/graphium
- Repository: https://github.com/datamol-io/graphium

## Overview

##### Project type and purpose
- This is a Python-based deep learning library (“Graphium”) for molecular graph representation learning.
- It provides:
- State-of-the-art GNN architectures (in `graphium/nn`).
- Rich featurization pipeline (in `graphium/features`).
- A PyTorch-Lightning‐based training loop and CLI (`graphium/trainer`, `graphium/cli`).
- Hydra‐driven config management for experiments (`expts/hydra-configs`).
- IPU support and optional precomputation (`graphium/ipu`, `graphium/data/datamodule`).

##### File structure
- Top-level folders:
- `graphium/`: core library modules (cli, config loader, data, features, nn, trainer, ipu utilities).
- `expts/`: Hydra experiment configs and scripts for running/train/predict.
- `tests/`: end-to-end and unit tests validating each component.
- Key patterns:
- Modular layered architecture (data → features → model → trainer).
- Hydra for config composition.
- PyTorch Lightning to isolate training logic.
- Plugin style CLI via entrypoints (`graphium/cli/__main__.py`).

##### Extract insights from README
- CLI entrypoint: `graphium` for data management, and `graphium-train` for training/inference.
- Data pipeline: “Graphium data prepare” for caching featurized graphs, then “graphium-train” reads preprocessed data.

##### Main components
a. CLI Layer
- graphium CLI (data commands)
- graphium-train CLI (train, finetune, predict)
b. Config Layer (Hydra)
- config loader (`graphium/config/*`)
- experiment configs (`expts/hydra-configs`)
c. Data Pipeline
- Raw data readers (`graphium/data/dataset.py`)
- Featurizer (`graphium/features/*`)
- DataModule (`graphium/data/datamodule.py`) & PyG batching
- Optional IPU data wrapper (`graphium/ipu/ipu_dataloader.py`)
d. Model Layer
- GNN architectures and encoders (`graphium/nn/architectures`, `graphium/nn/encoders`, `graphium/nn/pyg_layers`)
- Finetuning wrappers (`graphium/finetuning/*`)
e. Training & Evaluation
- PyTorch Lightning Trainer wrapper (`graphium/trainer`)
- Losses, metrics, predictors, hyperparam search (`graphium/trainer/*`, `graphium/hyper_param_search`)
- IPU training support (`graphium/ipu/ipu_simple_lightning.py`)
f. Outputs
- Checkpoints, logs, metrics, cached data
g. External Services & Dependencies
- PyTorch, PyTorch-Lightning, Hydra, Torch-Geometric, Graphcore IPU SDK

##### Relationships and interactions
- CLI invokes Config Loader → instantiates DataModule & Model & Trainer.
- DataModule fetches raw data → passes through Featurizer → batches into Graph objects.
- Model consumes batched Graphs → produces predictions.
- Trainer orchestrates forward, backward, optimizer steps, metrics → writes checkpoints & logs.
- Hyperparam Search hooks into Trainer to sample configs.
- IPU wrapper intercepts DataLoader and Trainer loops if accelerator=ipu.
- Finetuning reuses pretrained Model + custom TaskHead.

##### Architectural patterns & design principles
- Layered architecture (separate data, model, training).
- Plugin/config‐driven (Hydra, entrypoints).
- Extensible API (users can add new GNN layers in `graphium/nn/pyg_layers`).
- Caching & reproducibility (data preparation pipeline with hash‐based dirs).
- Hardware abstraction for accelerators (CPU/GPU/IPU).

##### What
- Clear labels: “graphium-train CLI”, “Hydra Config Loader”, “Graphium DataModule”, “GNN Architectures”, “Lightning Trainer”, “Checkpoint & Metrics”.
- Optional inset: IPU branch (highlight code in `graphium/ipu`).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        direction TB
        CLI_Data["graphium CLI\n(data commands)"]:::coreLib
        CLI_Train["graphium-train CLI\n(train/finetune/predict)"]:::coreLib
    end

    %% Config Layer
    subgraph "Config Layer (Hydra)"
        direction TB
        Config_Loader["Hydra Config Loader\n(graphium/config)"]:::coreLib
        Exp_Configs["Experiment Configs\n(expts/hydra-configs)"]:::exp
    end

    %% Data Pipeline
    subgraph "Data Pipeline"
        direction TB
        Raw_Data["Raw Data Readers\n(graphium/data/dataset.py)"]:::coreLib
        Featurizer["Featurization\n(graphium/features)"]:::coreLib
        DataModule["Graphium DataModule\n(graphium/data/datamodule.py)"]:::coreLib
        IPU_DL["IPU Data Wrapper\n(graphium/ipu/ipu_dataloader.py)"]:::coreLib
    end

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        GNN_Arch["GNN Architectures\n(graphium/nn/architectures)"]:::coreLib
        Encoders["Encoders\n(graphium/nn/encoders)"]:::coreLib
        PyG_Layers["PyG Layer Impl.\n(graphium/nn/pyg_layers)"]:::coreLib
        Finetuning["Finetuning Wrappers\n(graphium/finetuning)"]:::coreLib
    end

    %% Training & Evaluation
    subgraph "Training & Evaluation"
        direction TB
        Lightning_Trainer["Lightning Trainer Wrapper\n(graphium/trainer)"]:::coreLib
        HyperParam["Hyperparam Search\n(graphium/hyper_param_search)"]:::coreLib
        IPU_Trainer["IPU Training Support\n(graphium/ipu/ipu_simple_lightning.py)"]:::coreLib
    end

    %% Outputs
    subgraph "Outputs"
        direction TB
        Checkpoints["Checkpoints\n& Models"]:::coreLib
        Logs["Logs"]:::coreLib
        Metrics["Metrics"]:::coreLib
        Cached_Data["Cached Data"]:::coreLib
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        PyTorch["PyTorch"]:::ext
        Lightning["PyTorch-Lightning"]:::ext
        Hydra["Hydra"]:::ext
        PyG["Torch-Geometric"]:::ext
        IPU_SDK["Graphcore IPU SDK"]:::ext
    end

    %% Connections
    CLI_Data -->|uses| Config_Loader
    CLI_Train -->|uses| Config_Loader
    Config_Loader -->|loads| Exp_Configs

    Config_Loader -->|"instantiates"| DataModule
    Config_Loader -->|"instantiates"| GNN_Arch
    Config_Loader -->|"instantiates"| Lightning_Trainer

    DataModule -->|reads| Raw_Data
    Raw_Data -->|sends data to| Featurizer
    Featurizer -->|outputs| DataModule
    DataModule -->|batches| GNN_Arch
    DataModule -->|via IPU wrapper| IPU_DL
    IPU_DL -->|to| GNN_Arch

    GNN_Arch -->|calls| Encoders
    GNN_Arch -->|calls| PyG_Layers
    Encoders -->|optional| Finetuning
    PyG_Layers -->|optional| Finetuning

    GNN_Arch -->|forward pass| Lightning_Trainer
    Finetuning -->|forward pass| Lightning_Trainer
    Lightning_Trainer -->|evaluates| HyperParam
    Lightning_Trainer -->|uses| IPU_Trainer

    Lightning_Trainer -->|writes| Checkpoints
    Lightning_Trainer -->|writes| Logs
    Lightning_Trainer -->|writes| Metrics
    DataModule -->|caches| Cached_Data

    %% External Dependencies connections
    GNN_Arch -->|depends on| PyTorch
    Lightning_Trainer -->|depends on| Lightning
    Config_Loader -->|depends on| Hydra
    DataModule -->|depends on| PyG
    IPU_DL -->|depends on| IPU_SDK
    IPU_Trainer -->|depends on| IPU_SDK

    %% Click Events - CLI Layer
    click CLI_Data "https://github.com/datamol-io/graphium/blob/main/graphium/cli/data.py"
    click CLI_Train "https://github.com/datamol-io/graphium/blob/main/graphium/cli/train_finetune_test.py"
    %% Additional CLI utilities
    click CLI_Data "https://github.com/datamol-io/graphium/blob/main/graphium/cli/main.py"
    click CLI_Data "https://github.com/datamol-io/graphium/blob/main/graphium/cli/parameters.py"
    click CLI_Data "https://github.com/datamol-io/graphium/blob/main/graphium/cli/finetune_utils.py"
    click CLI_Data "https://github.com/datamol-io/graphium/blob/main/graphium/cli/fingerprints.py"
    click CLI_Data "https://github.com/datamol-io/graphium/blob/main/graphium/cli/__main__.py"

    %% Click Events - Config Layer
    click Config_Loader "https://github.com/datamol-io/graphium/blob/main/graphium/config/_loader.py"
    click Config_Loader "https://github.com/datamol-io/graphium/blob/main/graphium/config/_load.py"
    click Config_Loader "https://github.com/datamol-io/graphium/blob/main/graphium/config/config_convert.py"
    click Config_Loader "https://github.com/datamol-io/graphium/blob/main/graphium/config/config.yaml"
    click Exp_Configs "https://github.com/datamol-io/graphium/tree/main/expts/hydra-configs/"

    %% Click Events - Data Pipeline
    click Raw_Data "https://github.com/datamol-io/graphium/blob/main/graphium/data/dataset.py"
    click Featurizer "https://github.com/datamol-io/graphium/blob/main/graphium/features/featurizer.py"
    click Featurizer "https://github.com/datamol-io/graphium/blob/main/graphium/features/commute.py"
    click Featurizer "https://github.com/datamol-io/graphium/blob/main/graphium/features/electrostatic.py"
    click Featurizer "https://github.com/datamol-io/graphium/blob/main/graphium/features/positional_encoding.py"
    click DataModule "https://github.com/datamol-io/graphium/blob/main/graphium/data/datamodule.py"
    click DataModule "https://github.com/datamol-io/graphium/blob/main/graphium/data/collate.py"
    click IPU_DL "https://github.com/datamol-io/graphium/blob/main/graphium/ipu/ipu_dataloader.py"

    %% Click Events - Model Layer
    click GNN_Arch "https://github.com/datamol-io/graphium/blob/main/graphium/nn/architectures/global_architectures.py"
    click GNN_Arch "https://github.com/datamol-io/graphium/blob/main/graphium/nn/architectures/pyg_architectures.py"
    click GNN_Arch "https://github.com/datamol-io/graphium/blob/main/graphium/nn/architectures/encoder_manager.py"
    click Encoders "https://github.com/datamol-io/graphium/blob/main/graphium/nn/encoders/base_encoder.py"
    click PyG_Layers "https://github.com/datamol-io/graphium/blob/main/graphium/nn/pyg_layers/gcn_pyg.py"
    click PyG_Layers "https://github.com/datamol-io/graphium/blob/main/graphium/nn/pyg_layers/gin_pyg.py"
    click PyG_Layers "https://github.com/datamol-io/graphium/blob/main/graphium/nn/pyg_layers/mpnn_pyg.py"
    click Finetuning "https://github.com/datamol-io/graphium/blob/main/graphium/finetuning/finetuning.py"
    click Finetuning "https://github.com/datamol-io/graphium/blob/main/graphium/finetuning/finetuning_architecture.py"
    click Finetuning "https://github.com/datamol-io/graphium/blob/main/graphium/finetuning/fingerprinting.py"
    click Finetuning "https://github.com/datamol-io/graphium/blob/main/graphium/finetuning/utils.py"

    %% Click Events - Training & Evaluation
    click Lightning_Trainer "https://github.com/datamol-io/graphium/blob/main/graphium/trainer/__init__.py"
    click Lightning_Trainer "https://github.com/datamol-io/graphium/blob/main/graphium/trainer/losses.py"
    click Lightning_Trainer "https://github.com/datamol-io/graphium/blob/main/graphium/trainer/metrics.py"
    click Lightning_Trainer "https://github.com/datamol-io/graphium/blob/main/graphium/trainer/predictor.py"
    click Lightning_Trainer "https://github.com/datamol-io/graphium/blob/main/graphium/trainer/predictor_options.py"
    click Lightning_Trainer "https://github.com/datamol-io/graphium/blob/main/graphium/trainer/predictor_summaries.py"
    click HyperParam "https://github.com/datamol-io/graphium/blob/main/graphium/hyper_param_search/results.py"
    click IPU_Trainer "https://github.com/datamol-io/graphium/blob/main/graphium/ipu/ipu_simple_lightning.py"
    click IPU_Trainer "https://github.com/datamol-io/graphium/blob/main/graphium/ipu/ipu_losses.py"
    click IPU_Trainer "https://github.com/datamol-io/graphium/blob/main/graphium/ipu/ipu_metrics.py"
    click IPU_Trainer "https://github.com/datamol-io/graphium/blob/main/graphium/ipu/ipu_wrapper.py"
    click IPU_Trainer "https://github.com/datamol-io/graphium/blob/main/graphium/ipu/ipu_utils.py"

    %% Click Events - Experiment Entry Scripts
    click CLI_Train "https://github.com/datamol-io/graphium/blob/main/expts/main_run_multitask.py"
    click CLI_Train "https://github.com/datamol-io/graphium/blob/main/expts/main_run_predict.py"
    click CLI_Train "https://github.com/datamol-io/graphium/blob/main/expts/main_run_test.py"
    click CLI_Train "https://github.com/datamol-io/graphium/blob/main/expts/main_run_get_fingerprints.py"

    %% Click Events - Tests
    click Raw_Data "https://github.com/datamol-io/graphium/blob/main/tests/test_datamodule.py"
    click Raw_Data "https://github.com/datamol-io/graphium/blob/main/tests/test_dataset.py"
    click Featurizer "https://github.com/datamol-io/graphium/blob/main/tests/test_featurizer.py"
    click GNN_Arch "https://github.com/datamol-io/graphium/blob/main/tests/test_architectures.py"
    click PyG_Layers "https://github.com/datamol-io/graphium/blob/main/tests/test_pyg_layers.py"
    click Lightning_Trainer "https://github.com/datamol-io/graphium/blob/main/tests/test_trainer.py"
    click IPU_DL "https://github.com/datamol-io/graphium/blob/main/tests/test_ipu_dataloader.py"

    %% Styles
    classDef coreLib fill:#bbe1fa,stroke:#1f78b4,color:#000
    classDef exp fill:#c1f0c1,stroke:#2d8c2d,color:#000
    classDef ext fill:#ffdca8,stroke:#b35f00,color:#000
```
