# metatensor/metatrain architecture

> Architecture diagram of the GitHub repository metatensor/metatrain, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/metatensor/metatrain
- Repository: https://github.com/metatensor/metatrain

## Overview

##### Project type and purpose
- metatrain is an open-source command-line interface (CLI) tool for training, evaluating, and exporting atomistic ML models.
- It provides a unified YAML-based configuration, a plugin architecture for multiple “architectures” (GAP, PET, NanoPET, SOAP-BPNN), and generates standalone TorchScript models for use in MD engines (LAMMPS, i-PI, ASE).

##### File structure
- Top-level:
- src/metatrain: core Python package
– __main__.py, cli/ (train, eval, export, formatter), trainer modules, plugin packages (gap, pet, soap_bpnn, experimental/deprecated)
– utils/: configuration loader (omegaconf), data loaders, distributed helpers, logging, IO, metrics, etc.
- examples/, docs/, tests/: usage examples, Sphinx documentation, test suites
- CI and packaging files (.github/workflows/, tox.ini, pyproject.toml, README.rst)
- Patterns:
- CLI layer calls into a core “engine” (trainer, evaluator, exporter)
- Plugin modules implement a standardized interface (schema-hypers.json, default-hypers.yaml, model.py, trainer.py)
- Utilities factor out cross-cutting concerns: config parsing, data I/O, logging, distributed training

##### README for additional insights
- The tool revolves around a single CLI entrypoint: `mtt`
- Supports subcommands: train, evaluate, export, formatter
- Any TorchScript-compatible architecture can be dropped in as a plugin
- Outputs a TorchScript artifact for downstream MD engines

a. Main components
1. CLI Layer (“mtt” entrypoint)
2. Configuration Manager (OmegaConf-based YAML loader)
3. Core Engine
– Trainer
– Evaluator
– Exporter
4. Plugin System (architecture packages: gap, pet, soap_bpnn, experimental/deprecated)
5. Data I/O
– Dataset readers (ASE, Metatensor, generic XYZ)
– Data split (train/val/test)
6. Model Abstraction
– Hyperschema loader
– Model builder (PyTorch modules)
7. Utilities
– Logging
– Metrics
– Distributed training helpers
8. External Integrations
– TorchScript exporter
– MD engines (ASE, LAMMPS, i-PI)
b. Relationships & interactions
- User invokes `mtt <subcommand>` → CLI Layer
- CLI parses arguments → Configuration Manager → passes config object to Core Engine
- Core Engine calls Data I/O to load/split datasets
- Core Engine loads selected Plugin (architecture package) via the plugin interface
- Plugin provides:
– default hypers
– model definition (PyTorch)
– trainer logic (loss, optimizer, epochs)
- Trainer trains model, logs metrics (via Logging & Metrics utilities), optionally uses Distributed utilities
- Evaluator runs inference on validation/test sets
- Exporter serializes trained model to TorchScript
- TorchScript artifact is consumed by external MD engines at runtime
c. Architectural patterns
- Plugin pattern for architectures
- Layered CLI → Application Core → Plugin
- Separation of concerns: config, I/O, training logic, model definitions
- Use of JSON schema for config validation
d. Relevant technologies
- Python 3, PyTorch, TorchScript
- OmegaConf for YAML configuration
- Sphinx for documentation
- GitHub Actions for CI
- pytest for testing

##### Detail
- Every major package from src/metatrain (cli/, utils/, each architecture)
- How config schemas (schema-hypers.json) relate to the Configuration Manager
- If space permits, indicate CI/testing/documentation pipelines as separate clusters

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        direction TB
        cliMain["__main__.py"]:::frontend
        cliTrain["train.py"]:::frontend
        cliEval["eval.py"]:::frontend
        cliExport["export.py"]:::frontend
        cliFmt["formatter.py"]:::frontend
    end

    %% Configuration Manager
    subgraph "Configuration Manager"
        direction TB
        cfgOmegaconf["omegaconf.py"]:::core
        cfgJsonschema["jsonschema.py"]:::core
        schemaBase["schema-base.json"]:::core
        schemaDataset["schema-dataset.json"]:::core
    end

    %% Core Engine
    coreEngine["Core Engine"]:::core

    %% Data I/O
    subgraph "Data I/O"
        direction TB
        readers["readers/*.py"]:::core
        writers["writers/*.py"]:::core
        dsOrch["dataset orchestration"]:::core
    end

    %% Plugin System
    subgraph "Plugin System"
        direction TB
        gapMod["gap/model.py"]:::plugin
        gapTr["gap/trainer.py"]:::plugin
        gapHypDefaults["gap/default-hypers.yaml"]:::plugin
        gapHypSchema["gap/schema-hypers.json"]:::plugin

        petMod["pet/model.py"]:::plugin
        petTr["pet/trainer.py"]:::plugin
        petHypDefaults["pet/default-hypers.yaml"]:::plugin
        petHypSchema["pet/schema-hypers.json"]:::plugin

        soapMod["soap_bpnn/model.py"]:::plugin
        soapTr["soap_bpnn/trainer.py"]:::plugin
        soapHypDefaults["soap_bpnn/default-hypers.yaml"]:::plugin
        soapHypSchema["soap_bpnn/schema-hypers.json"]:::plugin

        nanoMod["experimental/nanopet/model.py"]:::plugin
        nanoTr["experimental/nanopet/trainer.py"]:::plugin
        nanoHypDefaults["experimental/nanopet/default-hypers.yaml"]:::plugin
        nanoHypSchema["experimental/nanopet/schema-hypers.json"]:::plugin

        depMod["deprecated/pet/model.py"]:::plugin
        depTr["deprecated/pet/trainer.py"]:::plugin
        depHypDefaults["deprecated/pet/default-hypers.yaml"]:::plugin
        depHypSchema["deprecated/pet/schema-hypers.json"]:::plugin
    end

    %% Utilities
    subgraph "Utilities"
        direction TB
        utilLog["logging.py"]:::util
        utilMetrics["metrics.py"]:::util
        utilDist["distributed/*.py"]:::util
        utilOther["cross-cutting utils"]:::util
    end

    %% Export & External
    exporter["Exporter"]:::core
    torchscript["TorchScript Artifact"]:::external
    subgraph "MD Engines"
        direction TB
        mdAse["ASE"]:::external
        mdLammps["LAMMPS"]:::external
        mdiPi["i-PI"]:::external
    end

    %% Connections
    cliMain -->|invokes| cliTrain
    cliMain --> cliEval
    cliMain --> cliExport
    cliMain --> cliFmt

    cliTrain -->|"load config"| cfgOmegaconf
    cliEval --> cfgOmegaconf
    cliExport --> cfgOmegaconf

    cfgOmegaconf --> cfgJsonschema
    cfgJsonschema --> schemaBase
    cfgJsonschema --> schemaDataset

    cliTrain --> coreEngine
    cliEval --> coreEngine
    cliExport --> coreEngine

    coreEngine --> dataIOReaders["Data Readers"] 
    coreEngine --> dataIOWriters["Data Writers"]
    coreEngine --> dsOrch

    coreEngine -->|"select plugin"| gapMod
    coreEngine --> petMod
    coreEngine --> soapMod
    coreEngine --> nanoMod
    coreEngine --> depMod

    gapMod --> gapTr
    petMod --> petTr
    soapMod --> soapTr
    nanoMod --> nanoTr
    depMod --> depTr

    gapTr --> utilLog
    petTr --> utilLog
    soapTr --> utilLog
    nanoTr --> utilLog
    depTr --> utilLog

    utilLog --> utilMetrics
    utilMetrics --> utilDist
    utilDist --> coreEngine

    coreEngine --> exporter
    exporter --> torchscript
    torchscript --> mdAse
    torchscript --> mdLammps
    torchscript --> mdiPi

    %% Click Events
    click cliMain "https://github.com/metatensor/metatrain/blob/main/src/metatrain/__main__.py"
    click cliTrain "https://github.com/metatensor/metatrain/blob/main/src/metatrain/cli/train.py"
    click cliEval "https://github.com/metatensor/metatrain/blob/main/src/metatrain/cli/eval.py"
    click cliExport "https://github.com/metatensor/metatrain/blob/main/src/metatrain/cli/export.py"
    click cliFmt "https://github.com/metatensor/metatrain/blob/main/src/metatrain/cli/formatter.py"

    click cfgOmegaconf "https://github.com/metatensor/metatrain/blob/main/src/metatrain/utils/omegaconf.py"
    click cfgJsonschema "https://github.com/metatensor/metatrain/blob/main/src/metatrain/utils/jsonschema.py"
    click schemaBase "https://github.com/metatensor/metatrain/blob/main/src/metatrain/share/schema-base.json"
    click schemaDataset "https://github.com/metatensor/metatrain/blob/main/src/metatrain/share/schema-dataset.json"

    click gapMod "https://github.com/metatensor/metatrain/blob/main/src/metatrain/gap/model.py"
    click gapTr "https://github.com/metatensor/metatrain/blob/main/src/metatrain/gap/trainer.py"
    click gapHypDefaults "https://github.com/metatensor/metatrain/blob/main/src/metatrain/gap/default-hypers.yaml"
    click gapHypSchema "https://github.com/metatensor/metatrain/blob/main/src/metatrain/gap/schema-hypers.json"

    click petMod "https://github.com/metatensor/metatrain/blob/main/src/metatrain/pet/model.py"
    click petTr "https://github.com/metatensor/metatrain/blob/main/src/metatrain/pet/trainer.py"
    click petHypDefaults "https://github.com/metatensor/metatrain/blob/main/src/metatrain/pet/default-hypers.yaml"
    click petHypSchema "https://github.com/metatensor/metatrain/blob/main/src/metatrain/pet/schema-hypers.json"

    click soapMod "https://github.com/metatensor/metatrain/blob/main/src/metatrain/soap_bpnn/model.py"
    click soapTr "https://github.com/metatensor/metatrain/blob/main/src/metatrain/soap_bpnn/trainer.py"
    click soapHypDefaults "https://github.com/metatensor/metatrain/blob/main/src/metatrain/soap_bpnn/default-hypers.yaml"
    click soapHypSchema "https://github.com/metatensor/metatrain/blob/main/src/metatrain/soap_bpnn/schema-hypers.json"

    click nanoMod "https://github.com/metatensor/metatrain/blob/main/src/metatrain/experimental/nanopet/model.py"
    click nanoTr "https://github.com/metatensor/metatrain/blob/main/src/metatrain/experimental/nanopet/trainer.py"
    click nanoHypDefaults "https://github.com/metatensor/metatrain/blob/main/src/metatrain/experimental/nanopet/default-hypers.yaml"
    click nanoHypSchema "https://github.com/metatensor/metatrain/blob/main/src/metatrain/experimental/nanopet/schema-hypers.json"

    click depMod "https://github.com/metatensor/metatrain/blob/main/src/metatrain/deprecated/pet/model.py"
    click depTr "https://github.com/metatensor/metatrain/blob/main/src/metatrain/deprecated/pet/trainer.py"
    click depHypDefaults "https://github.com/metatensor/metatrain/blob/main/src/metatrain/deprecated/pet/default-hypers.yaml"
    click depHypSchema "https://github.com/metatensor/metatrain/blob/main/src/metatrain/deprecated/pet/schema-hypers.json"

    click readers "https://github.com/metatensor/metatrain/tree/main/src/metatrain/utils/data/readers"
    click writers "https://github.com/metatensor/metatrain/tree/main/src/metatrain/utils/data/writers"
    click dsOrch "https://github.com/metatensor/metatrain/blob/main/src/metatrain/utils/data/dataset.py"
    click dataIOReaders "https://github.com/metatensor/metatrain/tree/main/src/metatrain/utils/data/readers"
    click dataIOWriters "https://github.com/metatensor/metatrain/tree/main/src/metatrain/utils/data/writers"

    click utilLog "https://github.com/metatensor/metatrain/blob/main/src/metatrain/utils/logging.py"
    click utilMetrics "https://github.com/metatensor/metatrain/blob/main/src/metatrain/utils/metrics.py"
    click utilDist "https://github.com/metatensor/metatrain/blob/main/src/metatrain/utils/distributed/distributed_data_parallel.py"

    click exporter "https://github.com/metatensor/metatrain/blob/main/src/metatrain/cli/export.py"

    %% Styles
    classDef frontend fill:#1f77b4,stroke:#000,color:#fff
    classDef core fill:#2ca02c,stroke:#000,color:#fff
    classDef plugin fill:#ff7f0e,stroke:#000,color:#000
    classDef util fill:#7f7f7f,stroke:#000,color:#fff
    classDef external fill:#9467bd,stroke:#000,color:#fff
```
