How metatensor/metatrain works
Project type and purpose
• metatrain is an open-source command-line interface (CLI) tool for training, evaluating, and exporting atomistic ML models.
• It provides a unified YAML-based configuration, a plugin architecture for multiple “architectures” (GAP, PET, NanoPET, SOAP-BPNN), and generates standalone TorchScript models for use in MD engines (LAMMPS, i-PI, ASE).
File structure
• Top-level:
• src/metatrain: core Python package
– __main__.py, cli/ (train, eval, export, formatter), trainer modules, plugin packages (gap, pet, soap_bpnn, experimental/deprecated)
– utils/: configuration loader (omegaconf), data loaders, distributed helpers, logging, IO, metrics, etc.
• examples/, docs/, tests/: usage examples, Sphinx documentation, test suites
• CI and packaging files (.github/workflows/, tox.ini, pyproject.toml, README.rst)
• Patterns:
• CLI layer calls into a core “engine” (trainer, evaluator, exporter)
• Plugin modules implement a standardized interface (schema-hypers.json, default-hypers.yaml, model.py, trainer.py)
• Utilities factor out cross-cutting concerns: config parsing, data I/O, logging, distributed training
README for additional insights
• The tool revolves around a single CLI entrypoint: mtt
• Supports subcommands: train, evaluate, export, formatter
• Any TorchScript-compatible architecture can be dropped in as a plugin
• Outputs a TorchScript artifact for downstream MD engines
a. Main components
1. CLI Layer (“mtt” entrypoint)
2. Configuration Manager (OmegaConf-based YAML loader)
3. Core Engine
– Trainer
– Evaluator
– Exporter
4. Plugin System (architecture packages: gap, pet, soap_bpnn, experimental/deprecated)
5. Data I/O
– Dataset readers (ASE, Metatensor, generic XYZ)
– Data split (train/val/test)
6. Model Abstraction
– Hyperschema loader
– Model builder (PyTorch modules)
7. Utilities
– Logging
– Metrics
– Distributed training helpers
8. External Integrations
– TorchScript exporter
– MD engines (ASE, LAMMPS, i-PI)
b. Relationships & interactions
• User invokes mtt <subcommand> → CLI Layer
• CLI parses arguments → Configuration Manager → passes config object to Core Engine
• Core Engine calls Data I/O to load/split datasets
• Core Engine loads selected Plugin (architecture package) via the plugin interface
• Plugin provides:
– default hypers
– model definition (PyTorch)
– trainer logic (loss, optimizer, epochs)
• Trainer trains model, logs metrics (via Logging & Metrics utilities), optionally uses Distributed utilities
• Evaluator runs inference on validation/test sets
• Exporter serializes trained model to TorchScript
• TorchScript artifact is consumed by external MD engines at runtime
c. Architectural patterns
• Plugin pattern for architectures
• Layered CLI → Application Core → Plugin
• Separation of concerns: config, I/O, training logic, model definitions
• Use of JSON schema for config validation
d. Relevant technologies
• Python 3, PyTorch, TorchScript
• OmegaConf for YAML configuration
• Sphinx for documentation
• GitHub Actions for CI
• pytest for testing
Detail
• Every major package from src/metatrain (cli/, utils/, each architecture)
• How config schemas (schema-hypers.json) relate to the Configuration Manager
• If space permits, indicate CI/testing/documentation pipelines as separate clusters