How kempnerinstitute/tmrc works
Project Type and Purpose
• TMRC is a research‐focused Python library and CLI for training transformer models.
• It provides a modular core for model definition, training loops, data handling, evaluation, and logging, geared toward single‐node and distributed HPC environments (Slurm, DDP/FSDP).
High-Level Architecture and Components
Major moving parts before drawing:
a. Configuration Layer
• YAML files under configs/training/ (Hydra/OmegaConf-style)
• CLI flags in train.py to select or override configs
b. Core Library (src/tmrc/core)
• Models: gpt.py + submodules/components (decoder, optimizers, quantizer)
• Training: train.py, data.py (data loading, batching, distributed sampler)
• Evaluation: metrics, scoring under core/evaluation
• Utilities: cluster launch (cluster.py), platform detection (platform.py), registry, stats
• Logging config (logging_config.py) sets up Python logging + Weights & Biases
c. Entry-point Scripts (scripts/)
• run_train.py: wrapper for local single‐GPU runs
• dist_slurm.sh: Slurm batch script for multi-node experiments
d. External Dependencies and Services
• PyTorch (DDP, FSDP) for model training and GPU utilization
• Hydra/OmegaConf for structured configs
• Weights & Biases for experiment tracking
• Slurm (on HPC cluster) or local shell for job scheduling
e. Documentation & Testing
• Sphinx docs under docs/ for user guide and API reference
Relationships and Data Flow
• User provides a config → CLI (train.py) loads config via Hydra →
Data module reads dataset paths →
Model module builds transformer from components →
Training loop dispatches to single‐GPU or DDP/FSDP across nodes →
Logging hooks send metrics to Weights & Biases →
Checkpoints written to disk.
• Scripts (dist_slurm.sh) invoke the same train.py entry point under srun.
• Tests import modules directly (core/models, core/training, core/evaluation) to verify behavior.
Architectural Patterns and Principles
• Modular design: clear separation of model, data, training, evaluation, and utilities
• Configuration‐driven: supports experimentation without code changes
• Pluggable components: optimizers, quantizers, decoders registered via a registry pattern
• Scalability: single‐node vs. multi-node DDP/FSDP strategies
Final Recommendations
• How scripts and HPC scheduler integrate with the same core entry point.
• Test harness and documentation as supporting elements, not part of the runtime.