# maincodehq/mainrun architecture

> Architecture diagram of the GitHub repository maincodehq/mainrun, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/maincodehq/mainrun
- Repository: https://github.com/maincodehq/mainrun

## Overview

##### Project Type and Purpose
- This is a single-repository, standardized ML training pipeline focused on training and optimizing a small GPT-2 style language model on a Hacker News headline dataset.
- It’s neither a web app nor microservices architecture; it’s a data-centric, experiment-driven Python codebase with a thin CLI layer for orchestration.

##### Core Components
a. Data Ingestion & Preparation
- mainrun/download_dataset.py
– Downloads or loads the raw dataset.
- mainrun/utils.py
– Contains tokenization, dataset splitting (train/validation), and any auxiliary preprocessing functions.
b. Model Definition & Training Loop
- mainrun/train.py
– Defines the transformer model architecture.
– Sets up the optimizer (e.g., Adam/Warmup), scheduler, loss function, random seed.
– Implements the training loop for 7 epochs, calls evaluate() each epoch.
c. Evaluation & Logging
- mainrun/train.py → evaluate() (immutable per spec)
- mainrun/logs/baseline.log and other epoch logs
– Records training/validation loss per epoch.
d. Orchestration & CLI
- Taskfile.yml
– Defines “task train” and “task submit” commands.
- scripts/checkpoint.mjs
– Creates periodic model checkpoints before full training.
- scripts/submit.mjs
– Packages up the repo and logs into a submission archive and triggers upload.
e. Reporting
- The user-added report.pdf (placed in mainrun/ before “task submit”) documents changes and results.

##### Relationships & Workflow
[Dataset Source]
↓
[download_dataset.py]
↓
[Preprocessing / Tokenization (utils.py)]
↓
[DataLoader] → [Training Loop (train.py)] ← [Model Definition]
↓
[evaluate()] → [Validation Loss]
↓
[Logging → Logs Directory]
↘
[Checkpoint.mjs] (pre-training)
↓
[submit.mjs] (pack & upload)

##### Architectural Patterns & Principles
- Pipeline pattern: clear separation of data ingestion, preprocessing, training, and evaluation stages.
- Single-entry orchestration: All functionality is exposed via “task train” and “task submit” for reproducibility.
- Immutable evaluation: evaluate() is encapsulated and should not be modified (per rules).
- Modularity: utils.py for shared helpers, scripts folder for CLI tooling, mainrun for core logic.

##### Technologies & Frameworks
- Python 3.10 (main runtime)
- PyTorch 2.x (model & training)
- Taskfile (Make-like CLI orchestration)
- Node.js (for the .mjs checkpoint & submit scripts)
- Optional: VS Code Dev Container (irrelevant to code logic per instructions)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Prep
    subgraph "Data Preparation"
        A["Dataset Source"]:::external
        B["download_dataset.py"]:::python
        C["utils.py"]:::python
        D["DataLoader"]:::python
    end

    %% Core Training
    subgraph "Core Training"
        E["train.py"]:::python
        F["evaluate()"]:::python
        G["logs/"]:::python
    end

    %% CLI Orchestration
    subgraph "CLI Tooling"
        H["Taskfile.yml"]:::script
        I["checkpoint.mjs"]:::script
        J["submit.mjs"]:::script
    end

    %% Data Flow
    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> J

    %% Control Flow
    H -->|task train| B
    H -->|task train| I
    I -->|checkpoint before training| E
    H -->|task submit| J

    %% Click Events
    click B "https://github.com/maincodehq/mainrun/blob/main/mainrun/download_dataset.py"
    click C "https://github.com/maincodehq/mainrun/blob/main/mainrun/utils.py"
    click E "https://github.com/maincodehq/mainrun/blob/main/mainrun/train.py"
    click F "https://github.com/maincodehq/mainrun/blob/main/mainrun/train.py"
    click G "https://github.com/maincodehq/mainrun/tree/main/mainrun/logs/"
    click H "https://github.com/maincodehq/mainrun/blob/main/Taskfile.yml"
    click I "https://github.com/maincodehq/mainrun/blob/main/scripts/checkpoint.mjs"
    click J "https://github.com/maincodehq/mainrun/blob/main/scripts/submit.mjs"

    %% Styles
    classDef python fill:#BBDEFB,stroke:#333,stroke-width:1px
    classDef script fill:#FFCC80,stroke:#333,stroke-width:1px
    classDef external fill:#E0E0E0,stroke:#333,stroke-width:1px

    %% Link Styles
    linkStyle 0 stroke:#0f0,stroke-width:2px
    linkStyle 1 stroke:#0f0,stroke-width:2px
    linkStyle 2 stroke:#0f0,stroke-width:2px
    linkStyle 3 stroke:#0f0,stroke-width:2px
    linkStyle 4 stroke:#0f0,stroke-width:2px
    linkStyle 5 stroke:#0f0,stroke-width:2px
    linkStyle 6 stroke:#888,stroke-width:1.5px,stroke-dasharray: 2 2
    linkStyle 7 stroke:#888,stroke-width:1.5px,stroke-dasharray: 2 2
    linkStyle 8 stroke:#888,stroke-width:1.5px,stroke-dasharray: 2 2
    linkStyle 9 stroke:#888,stroke-width:1.5px,stroke-dasharray: 2 2
```
