How datamol-io/graphium works
Project type and purpose
• This is a Python-based deep learning library (“Graphium”) for molecular graph representation learning.
• It provides:
• State-of-the-art GNN architectures (in graphium/nn).
• Rich featurization pipeline (in graphium/features).
• A PyTorch-Lightning‐based training loop and CLI (graphium/trainer, graphium/cli).
• Hydra‐driven config management for experiments (expts/hydra-configs).
• IPU support and optional precomputation (graphium/ipu, graphium/data/datamodule).
File structure
• Top-level folders:
• graphium/: core library modules (cli, config loader, data, features, nn, trainer, ipu utilities).
• expts/: Hydra experiment configs and scripts for running/train/predict.
• tests/: end-to-end and unit tests validating each component.
• Key patterns:
• Modular layered architecture (data → features → model → trainer).
• Hydra for config composition.
• PyTorch Lightning to isolate training logic.
• Plugin style CLI via entrypoints (graphium/cli/__main__.py).
Extract insights from README
• CLI entrypoint: graphium for data management, and graphium-train for training/inference.
• Data pipeline: “Graphium data prepare” for caching featurized graphs, then “graphium-train” reads preprocessed data.
Main components
a. CLI Layer
• graphium CLI (data commands)
• graphium-train CLI (train, finetune, predict)
b. Config Layer (Hydra)
• config loader (graphium/config/*)
• experiment configs (expts/hydra-configs)
c. Data Pipeline
• Raw data readers (graphium/data/dataset.py)
• Featurizer (graphium/features/*)
• DataModule (graphium/data/datamodule.py) & PyG batching
• Optional IPU data wrapper (graphium/ipu/ipu_dataloader.py)
d. Model Layer
• GNN architectures and encoders (graphium/nn/architectures, graphium/nn/encoders, graphium/nn/pyg_layers)
• Finetuning wrappers (graphium/finetuning/*)
e. Training & Evaluation
• PyTorch Lightning Trainer wrapper (graphium/trainer)
• Losses, metrics, predictors, hyperparam search (graphium/trainer/*, graphium/hyper_param_search)
• IPU training support (graphium/ipu/ipu_simple_lightning.py)
f. Outputs
• Checkpoints, logs, metrics, cached data
g. External Services & Dependencies
• PyTorch, PyTorch-Lightning, Hydra, Torch-Geometric, Graphcore IPU SDK
Relationships and interactions
• CLI invokes Config Loader → instantiates DataModule & Model & Trainer.
• DataModule fetches raw data → passes through Featurizer → batches into Graph objects.
• Model consumes batched Graphs → produces predictions.
• Trainer orchestrates forward, backward, optimizer steps, metrics → writes checkpoints & logs.
• Hyperparam Search hooks into Trainer to sample configs.
• IPU wrapper intercepts DataLoader and Trainer loops if accelerator=ipu.
• Finetuning reuses pretrained Model + custom TaskHead.
Architectural patterns & design principles
• Layered architecture (separate data, model, training).
• Plugin/config‐driven (Hydra, entrypoints).
• Extensible API (users can add new GNN layers in graphium/nn/pyg_layers).
• Caching & reproducibility (data preparation pipeline with hash‐based dirs).
• Hardware abstraction for accelerators (CPU/GPU/IPU).
What
• Clear labels: “graphium-train CLI”, “Hydra Config Loader”, “Graphium DataModule”, “GNN Architectures”, “Lightning Trainer”, “Checkpoint & Metrics”.
• Optional inset: IPU branch (highlight code in graphium/ipu).