# kempnerinstitute/tmrc architecture

> Architecture diagram of the GitHub repository kempnerinstitute/tmrc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kempnerinstitute/tmrc
- Repository: https://github.com/kempnerinstitute/tmrc

## Overview

##### Project Type and Purpose
- TMRC is a research‐focused Python library and CLI for training transformer models.
- It provides a modular core for model definition, training loops, data handling, evaluation, and logging, geared toward single‐node and distributed HPC environments (Slurm, DDP/FSDP).

##### High-Level Architecture and Components
Major moving parts before drawing:
a. Configuration Layer
- YAML files under `configs/training/` (Hydra/OmegaConf-style)
- CLI flags in `train.py` to select or override configs
b. Core Library (`src/tmrc/core`)
- Models: `gpt.py` + submodules/components (decoder, optimizers, quantizer)
- Training: `train.py`, `data.py` (data loading, batching, distributed sampler)
- Evaluation: metrics, scoring under `core/evaluation`
- Utilities: cluster launch (`cluster.py`), platform detection (`platform.py`), registry, stats
- Logging config (`logging_config.py`) sets up Python logging + Weights & Biases
c. Entry-point Scripts (`scripts/`)
- `run_train.py`: wrapper for local single‐GPU runs
- `dist_slurm.sh`: Slurm batch script for multi-node experiments
d. External Dependencies and Services
- PyTorch (DDP, FSDP) for model training and GPU utilization
- Hydra/OmegaConf for structured configs
- Weights & Biases for experiment tracking
- Slurm (on HPC cluster) or local shell for job scheduling
e. Documentation & Testing
- Sphinx docs under `docs/` for user guide and API reference

##### Relationships and Data Flow
- User provides a config → CLI (`train.py`) loads config via Hydra →
Data module reads dataset paths →
Model module builds transformer from components →
Training loop dispatches to single‐GPU or DDP/FSDP across nodes →
Logging hooks send metrics to Weights & Biases →
Checkpoints written to disk.
- Scripts (`dist_slurm.sh`) invoke the same `train.py` entry point under `srun`.
- Tests import modules directly (core/models, core/training, core/evaluation) to verify behavior.

##### Architectural Patterns and Principles
- Modular design: clear separation of model, data, training, evaluation, and utilities
- Configuration‐driven: supports experimentation without code changes
- Pluggable components: optimizers, quantizers, decoders registered via a registry pattern
- Scalability: single‐node vs. multi-node DDP/FSDP strategies

##### Final Recommendations
- How scripts and HPC scheduler integrate with the same core entry point.
- Test harness and documentation as supporting elements, not part of the runtime.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Layer
    subgraph "Configuration Layer"
        ConfigYAML["YAML Configs\nconfigs/training/"]:::config
        CLI["CLI Flags\n(in overrides)"]:::config
    end

    %% Entry-point Scripts
    subgraph "Entry-point Scripts"
        LocalScript["run_train.py"]:::scripts
        SlurmScript["dist_slurm.sh"]:::scripts
    end

    %% Core Library
    subgraph "Core Library"
        subgraph Models
            GPT["gpt.py"]:::core
            Decoder["decoder.py"]:::core
            Optimizers["optimizers.py"]:::core
            Quantizer["quantizer.py"]:::core
        end
        subgraph Training
            TrainCore["train.py"]:::core
            Data["data.py"]:::core
        end
        subgraph Evaluation
            Eval["evaluation/"]:::core
        end
        subgraph Utils
            Cluster["cluster.py"]:::core
            Platform["platform.py"]:::core
            Registry["registry.py"]:::core
            Stats["stats.py"]:::core
        end
        LoggingConfig["logging_config.py"]:::core
    end

    %% External Systems
    subgraph "External Systems"
        PyTorch["PyTorch\n(DDP/FSDP)"]:::external
        Hydra["Hydra/OmegaConf"]:::external
        WandB["Weights & Biases"]:::external
        SlurmCluster["Slurm/HPC Cluster"]:::external
        FileSystem["File System"]:::external
    end

    %% Documentation & Tests
    subgraph "Documentation & Tests"
        Docs["docs/"]:::doc
        Tests["tests/"]:::doc
    end

    %% Connections
    ConfigYAML -->|provides config| TrainCore
    CLI -->|overrides| TrainCore
    LocalScript -->|calls entry point| TrainCore
    SlurmScript -->|srun invokes| TrainCore

    TrainCore -->|loads config via| Hydra
    TrainCore -->|builds model| GPT
    TrainCore -->|loads data| Data
    Data -->|batch of tokens| TrainCore

    GPT -->|uses component| Decoder
    GPT -->|uses component| Optimizers
    GPT -->|uses component| Quantizer

    TrainCore -->|dispatch to| PyTorch
    PyTorch -->|executes on| FileSystem
    TrainCore -->|writes checkpoints| FileSystem

    TrainCore -->|log metrics| WandB
    LoggingConfig -->|configures logging| TrainCore
    LoggingConfig -->|sends metrics| WandB

    SlurmScript -->|submits job to| SlurmCluster
    SlurmCluster -->|allocates nodes for| PyTorch

    Registry -->|registers components| GPT
    Registry -->|registers components| Data
    Stats -->|collects stats from| TrainCore
    Platform -->|detects environment for| TrainCore
    Cluster -->|launches cluster jobs| TrainCore

    TrainCore -->|evaluate with| Eval

    Tests -->|validate| GPT
    Tests -->|validate| Data
    Tests -->|validate| TrainCore

    Docs -->|document configs| ConfigYAML
    Docs -->|document models| GPT

    %% Click Events
    click ConfigYAML "https://github.com/kempnerinstitute/tmrc/blob/develop/configs/training/default_train_config.yaml"
    click TrainCore "https://github.com/kempnerinstitute/tmrc/blob/develop/src/tmrc/core/training/train.py"
    click LocalScript "https://github.com/kempnerinstitute/tmrc/blob/develop/scripts/run_train.py"
    click SlurmScript "https://github.com/kempnerinstitute/tmrc/blob/develop/scripts/dist_slurm.sh"
    click GPT "https://github.com/kempnerinstitute/tmrc/blob/develop/src/tmrc/core/models/gpt.py"
    click Decoder "https://github.com/kempnerinstitute/tmrc/blob/develop/src/tmrc/core/models/components/decoder.py"
    click Optimizers "https://github.com/kempnerinstitute/tmrc/blob/develop/src/tmrc/core/models/components/optimizers.py"
    click Quantizer "https://github.com/kempnerinstitute/tmrc/blob/develop/src/tmrc/core/models/components/quantizer.py"
    click Data "https://github.com/kempnerinstitute/tmrc/blob/develop/src/tmrc/core/training/data.py"
    click Eval "https://github.com/kempnerinstitute/tmrc/tree/develop/src/tmrc/core/evaluation/"
    click Cluster "https://github.com/kempnerinstitute/tmrc/blob/develop/src/tmrc/core/utils/cluster.py"
    click Platform "https://github.com/kempnerinstitute/tmrc/blob/develop/src/tmrc/core/utils/platform.py"
    click Registry "https://github.com/kempnerinstitute/tmrc/blob/develop/src/tmrc/core/utils/registry.py"
    click Stats "https://github.com/kempnerinstitute/tmrc/blob/develop/src/tmrc/core/utils/stats.py"
    click LoggingConfig "https://github.com/kempnerinstitute/tmrc/blob/develop/src/tmrc/logging_config.py"
    click Docs "https://github.com/kempnerinstitute/tmrc/tree/develop/docs/"
    click Tests "https://github.com/kempnerinstitute/tmrc/tree/develop/tests/"

    %% Styles
    classDef core fill:#c2f0c2,stroke:#333,stroke-width:1px
    classDef config fill:#ffd966,stroke:#333,stroke-width:1px
    classDef scripts fill:#f9cb9c,stroke:#333,stroke-width:1px
    classDef external fill:#9fc5e8,stroke:#333,stroke-width:1px,stroke-dasharray:4 2
    classDef doc fill:#cccccc,stroke:#333,stroke-width:1px,stroke-dasharray:2 2
```
