# mst272/llm-dojo architecture

> Architecture diagram of the GitHub repository mst272/llm-dojo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mst272/llm-dojo
- Repository: https://github.com/mst272/llm-dojo

## Overview

- Data & Configuration:
- Data Source: The “data” directory hosts JSONL files that supply SFT, RLHF, and potentially VLM training data.
- Configuration Files: There are Deepspeed configuration files (in “train_args/deepspeed_config” and “rlhf/ds_config”) and various argument scripts to configure the training (e.g., “train_args/common_args.py”, “rlhf/rlhf_args/…”).

- SFT Training Module:
- Core Training Script: “main_train.py” is the entry point for standard training.
- Training Arguments: “train_args” directory provides SFT-specific configurations, including scripts to adapt to chat templates and support changes (Lora, Qlora, etc.).
- Data Processing: “utils/script/generate_data.py” plays a role in adapting raw instruction-output files into the required format.

- RLHF Training Module:
- RLHF Framework: The “rlhf” directory encompasses the RLHF training pipeline, including methods like PPO, DPO, RLOO, and SimPO.
- Knowledge Distillation & Rejected Sampling: Subsections within “rlhf” such as “rejected_sampling” and the “train_rlhf.py” / “train_gkd.py” scripts illustrate the handling of various RLHF techniques.
- Argument Management: Similar to SFT, RLHF has its own set of configuration and argument files under “rlhf/rlhf_args”.

- Evaluation Module:
- Located under the “evaluate” directory, it contains scripts for setting up and running evaluations (e.g., “evaluation.py”, “execution.py”, and “dataset” with Humaneval examples).
- Task-specific Scripts: The “evaluate/task” and “evaluate/generate.py” modules inform how model outputs are tested.

- VLM (Visual Language Modeling) Module:
- Inference and Training: “inference_vllm.py” and “vlm_train.py” along with a “train_vlm_config” directory (under “train_args/vlm_config”) provide support for VLM tasks.
- Multi-modal Components: The README outlines VLM support (QA tasks using vision and text) with configuration options for freezing visual components.

- LLM Tricks & Experimental Implementations:
- The “llm_tricks” directory holds examples and experimental code for various advanced techniques. Each subfolder (e.g., “DPO_example”, “dora”, “moe”, “transformer”) represents independent experimental modules offering innovative training/trick approaches.

- Utility & Helper Components:
- “utils” contains helper functions for data collation, processing, evaluation utilities, and scripts for tasks like downloading models or merging LoRA files.

##### Relationships and Interactions
- That both the SFT and RLHF modules generate trained models that further feed into the Evaluation Module.
- That “llm_tricks” serves as a sandbox for experiments, which may also depend on similar data and configuration inputs.

##### Architectural Patterns and Principles
- Modularity: The clear separation between “sft”, “rlhf”, “vlm”, “evaluate”, and “llm_tricks” illustrates a modular architecture where each major training method or experiment encapsulates its logic.
- Configuration-Driven: Notice the use of parameter files and config scripts (Deepspeed YAML/JSON and common_args Python modules) that allow dynamic adjustments without deep code changes.
- Reusability: The use of “utils” and argument management directories across both SFT and RLHF promotes code reuse across modules.
- Extensibility: The “llm_tricks” submodules highlight an extensible design – each experimental method is isolated and independently documented.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Configuration
    subgraph "Data & Configuration"
        D1("Data Source: JSONL Files"):::dataStyle
        D2("Configuration: DS Config & Args"):::dataStyle
    end

    %% SFT Training Module
    subgraph "SFT Training Module"
        S1("Core Training Script (main_train.py)"):::sftStyle
        S2("SFT Arguments (train_args/sft)"):::sftStyle
        S3("Data Processing (generate_data.py)"):::sftStyle
    end

    %% RLHF Training Module
    subgraph "RLHF Training Module"
        R1("RLHF Trainer (train_rlhf.py / train_gkd.py)"):::rlhfStyle
        R2("RLHF Deepspeed Config (rlhf/ds_config)"):::rlhfStyle
        R3("RLHF Arguments (rlhf/rlhf_args)"):::rlhfStyle
        R4("Rejected Sampling Techniques"):::rlhfStyle
    end

    %% VLM Pipeline
    subgraph "VLM Pipeline"
        V1("Inference (inference_vllm.py)"):::vlmStyle
        V2("VLM Training (vlm_train.py)"):::vlmStyle
        V3("VLM Configuration (train_args/vlm_config)"):::vlmStyle
        V4("VLM Utilities (vlm_template.py & eval/vllm)"):::vlmStyle
    end

    %% Evaluation Module
    subgraph "Evaluation Module"
        E1("Evaluation Pipeline (evaluation.py, execution.py, generate.py)"):::evalStyle
        E2("Evaluation Datasets (evaluate/dataset & evaluate/task)"):::evalStyle
    end

    %% LLM Tricks & Experiments
    subgraph "LLM Tricks & Experiments"
        L1("DPO Experiments"):::tricksStyle
        L2("Dora Experiments"):::tricksStyle
        L3("MOE Experiments"):::tricksStyle
        L4("Transformer from-scratch"):::tricksStyle
    end

    %% Utility & Helper Components
    subgraph "Utility & Helper Components"
        U1("General Utilities (utils)"):::utilStyle
        U2("Evaluation Helpers (utils/eval)"):::utilStyle
        U3("Additional Scripts (utils/script)"):::utilStyle
    end

    %% Relationships from Data & Configuration to Training Modules and Experiments
    D1 -->|"provides JSONL data"| S1
    D1 -->|"provides JSONL data"| R1
    D1 -->|"provides JSONL data"| V1
    D1 -->|"supports experiments"| L1

    D2 -->|"supplies configs"| S2
    D2 -->|"supplies configs"| R3
    D2 -->|"supplies configs"| V3

    %% Utilities support Training Modules
    U1 -->|"supports data processing"| S3
    U1 -->|"supports RLHF techniques"| R4
    U3 -->|"assists training scripts"| S1
    U3 -->|"assists RLHF scripts"| R1
    U2 -->|"aids evaluation"| E1

    %% Training Modules produce models for evaluation
    S1 -->|"trained model"| E1
    R1 -->|"trained model"| E1
    V1 -->|"trained model"| E1

    %% Additional internal connections in RLHF Module
    R3 --> R1
    R2 --> R1
    R4 --> R1

    %% LLM Tricks dependencies (using similar data and configs)
    L1 -->|"experimentally uses"| D1
    L2 -->|"experimentally uses"| D2
    L3 -->|"experimentally uses"| D1
    L4 -->|"experimentally uses"| D2

    %% Click Events for Data & Configuration
    click D1 "https://github.com/mst272/llm-dojo/tree/main/data"
    click D2 "https://github.com/mst272/llm-dojo/tree/main/train_args/deepspeed_config"

    %% Click Events for SFT Training Module
    click S1 "https://github.com/mst272/llm-dojo/blob/main/main_train.py"
    click S2 "https://github.com/mst272/llm-dojo/tree/main/train_args/sft"
    click S3 "https://github.com/mst272/llm-dojo/blob/main/utils/script/generate_data.py"

    %% Click Events for RLHF Training Module
    click R1 "https://github.com/mst272/llm-dojo/blob/main/rlhf/train_rlhf.py"
    click R2 "https://github.com/mst272/llm-dojo/tree/main/rlhf/ds_config"
    click R3 "https://github.com/mst272/llm-dojo/tree/main/rlhf/rlhf_args"
    click R4 "https://github.com/mst272/llm-dojo/tree/main/rlhf/rejected_sampling"

    %% Click Events for VLM Pipeline
    click V1 "https://github.com/mst272/llm-dojo/blob/main/inference_vllm.py"
    click V2 "https://github.com/mst272/llm-dojo/blob/main/vlm_train.py"
    click V3 "https://github.com/mst272/llm-dojo/tree/main/train_args/vlm_config"
    click V4 "https://github.com/mst272/llm-dojo/blob/main/utils/vlm_template.py"

    %% Click Events for Evaluation Module
    click E1 "https://github.com/mst272/llm-dojo/tree/main/evaluate"
    click E2 "https://github.com/mst272/llm-dojo/tree/main/evaluate/dataset"

    %% Click Events for LLM Tricks & Experiments
    click L1 "https://github.com/mst272/llm-dojo/tree/main/llm_tricks/DPO_example"
    click L2 "https://github.com/mst272/llm-dojo/tree/main/llm_tricks/dora"
    click L3 "https://github.com/mst272/llm-dojo/tree/main/llm_tricks/moe"
    click L4 "https://github.com/mst272/llm-dojo/tree/main/llm_tricks/transformer"

    %% Click Events for Utility & Helper Components
    click U1 "https://github.com/mst272/llm-dojo/tree/main/utils"
    click U2 "https://github.com/mst272/llm-dojo/tree/main/utils/eval"
    click U3 "https://github.com/mst272/llm-dojo/tree/main/utils/script"

    %% Styles
    classDef dataStyle fill:#f2c9ac,stroke:#d16a16,stroke-width:2px;
    classDef sftStyle fill:#cce5ff,stroke:#007bff,stroke-width:2px;
    classDef rlhfStyle fill:#d4edda,stroke:#28a745,stroke-width:2px;
    classDef evalStyle fill:#fff3cd,stroke:#ffc107,stroke-width:2px;
    classDef vlmStyle fill:#f8d7da,stroke:#dc3545,stroke-width:2px;
    classDef tricksStyle fill:#d1ecf1,stroke:#17a2b8,stroke-width:2px;
    classDef utilStyle fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
```
