# jainilpatel2502/minireasoner-70m architecture

> Architecture diagram of the GitHub repository jainilpatel2502/minireasoner-70m, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jainilpatel2502/minireasoner-70m
- Repository: https://github.com/jainilpatel2502/minireasoner-70m

## Overview

##### Project type and purpose
- This is a **monolithic Python transformer modeling** project (educational full-stack pipeline).
- Purpose: implement, train, and serve a 70M-parameter transformer from scratch—including data ingestion, tokenizer training, model definition, training loop, checkpointing/logging, and inference scripts.

##### File structure
Top-level modules:
- MultiheadAttention.py, RMSNorm.py, SwiGLU.py, rope.py, rotatory_pos_embd.py
– low-level building blocks for attention, normalization, activations, positional embeddings
- TransformerBlock.py, Transformer.py
– assemble blocks into full model architecture
- training.py
– orchestrates data loading, forward/backward passes, optimizer, LR schedule, checkpointing, mixed precision
- sample_outputs.py
– inference script to generate text samples from a checkpoint
- app.py
– lightweight serving or interactive prompt wrapper around the model
- count_par.py
– utility to inspect model parameter counts
Data/ directory:
- data_collector_for_tokenization.py, tokenizer.py, DataLoader.py, data_fetcher.py, test_tokenizer.py
– build and exercise a custom tokenizer, streaming DataLoader that reads synthetic dataset
- tokenizer.model, tokenizer.vocab
– artifacts produced by tokenizer training

##### Extract README insights
- Tech stack: PyTorch, FlashAttention, bfloat16 mixed precision, TensorBoard
- Features: PreNorm design, SwiGLU, RMSNorm, RoPE, weight-tying, cosine LR decay, checkpoint every 500 steps
- Data origin: nvidia/OpenCodeReasoning synthetic tokens (external dataset)

##### Main components and interactions
A. Data Ingestion & Tokenizer Training
– External dataset (Hugging Face / local files) → data_fetcher.py → data_collector_for_tokenization.py
– tokenizer.py ← trains BPE/WordPiece model → outputs tokenizer.model & tokenizer.vocab
B. DataLoader
– streaming loader (DataLoader.py) uses tokenizer to convert raw text → token IDs in batches
C. Model Definition
– building blocks (MultiheadAttention, RMSNorm, SwiGLU, RoPE) → TransformerBlock → Transformer
D. Training Engine
– training.py:
· loads DataLoader batches → feeds into Transformer
· applies optimizer, gradient clipping, mixed precision (bfloat16)
· updates weights, logs metrics to TensorBoard
· checkpointing module saves model & optimizer states every N steps
E. Utilities
– count_par.py inspects total parameters
F. Inference / Serving
– sample_outputs.py: loads checkpoint, runs generate() with sampling settings
– app.py: wraps sampling in an interactive script or API endpoint

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Dataset
    ExtData["External Dataset<br>(nvidia/OpenCodeReasoning)"]:::external

    %% Data Ingestion & Tokenizer Training
    subgraph "Data Ingestion & Tokenizer Training"
        DF["data_fetcher.py"]:::compute
        DC["data_collector_for_tokenization.py"]:::compute
        TK["tokenizer.py"]:::compute
        TM["tokenizer.model"]:::storage
        TV["tokenizer.vocab"]:::storage
    end

    %% DataLoader
    subgraph "DataLoader"
        DL["DataLoader.py"]:::compute
        TT["test_tokenizer.py"]:::compute
    end

    %% Model Definition
    subgraph "Model Definition"
        MA["MultiheadAttention.py"]:::compute
        RN["RMSNorm.py"]:::compute
        SG["SwiGLU.py"]:::compute
        RO["rope.py"]:::compute
        RP["rotatory_pos_embd.py"]:::compute
        TB["TransformerBlock.py"]:::compute
        TR["Transformer.py"]:::compute
    end

    %% Training Engine
    subgraph "Training Engine"
        TE["training.py"]:::compute
    end

    %% Storage & Logging
    subgraph "Storage & Logging"
        CP["Checkpoints"]:::storage
        TBLog["TensorBoard Logs"]:::storage
    end

    %% External TensorBoard UI
    TBUI["TensorBoard/Web UI"]:::external

    %% Inference & Serving
    subgraph "Inference & Serving"
        SO["sample_outputs.py"]:::serving
        AP["app.py"]:::serving
    end

    %% Utilities
    subgraph "Utilities"
        CPAR["count_par.py"]:::serving
    end

    %% Data Flow
    ExtData -->|raw text| DF
    DF --> DC --> TK -->|tokenizer artifacts| TM & TV
    TM & TV -->|tokenizer artifacts| DL
    DL -->|token IDs| TE
    TE -->|model weights| CP
    TE -->|metrics| TBLog
    TBLog --> TBUI
    CP --> SO & AP
    TE -->|bfloat16 + grad clipping| TE
    SO -->|generated samples| AP

    %% Click Events
    click DF "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/Data/data_fetcher.py"
    click DC "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/Data/data_collector_for_tokenization.py"
    click TK "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/Data/tokenizer.py"
    click TM "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/Data/tokenizer.model"
    click TV "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/Data/tokenizer.vocab"
    click DL "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/Data/DataLoader.py"
    click TT "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/Data/test_tokenizer.py"
    click MA "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/MultiheadAttention.py"
    click RN "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/RMSNorm.py"
    click SG "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/SwiGLU.py"
    click RO "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/rope.py"
    click RP "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/rotatory_pos_embd.py"
    click TB "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/TransformerBlock.py"
    click TR "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/Transformer.py"
    click TE "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/training.py"
    click CPAR "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/count_par.py"
    click SO "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/sample_outputs.py"
    click AP "https://github.com/jainilpatel2502/minireasoner-70m/blob/main/app.py"

    %% Styles
    classDef external fill:#CCCCCC,stroke:#333,stroke-width:1px
    classDef storage fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef compute fill:#90EE90,stroke:#333,stroke-width:1px
    classDef serving fill:#FFA500,stroke:#333,stroke-width:1px
```
