# freedomintelligence/huatuogpt-o1 architecture

> Architecture diagram of the GitHub repository freedomintelligence/huatuogpt-o1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/freedomintelligence/huatuogpt-o1
- Repository: https://github.com/freedomintelligence/huatuogpt-o1

## Overview

##### Main Components

a) Training Pipeline (Two-Stage Architecture):
- Stage 1: Supervised Fine-Tuning (SFT) Component
- Stage 2: Reinforcement Learning (PPO) Component
- Medical Verifier Component
- Data Processing Pipeline

b) Model Variants:
- Multiple Model Implementations (7B, 8B, 70B, 72B)
- Different Backbone Models (LLaMA-3.1 and Qwen2.5)

##### Key Relationships to Highlight

a) Data Flow:
- Problem Construction → SFT Training → RL Training
- Verifier Feedback Loop in RL Stage
- Complex Reasoning Path Search Process

b) Training Process Flow:
- Data Processing → Model Training → Evaluation
- Reward Computation Flow in RL Stage

##### Specific Implementation Details

**[Data Processing Layer]**
- Verifiable Problem Constructor
- Complex Reasoning Path Searcher
- Training Data Pipeline

**[Model Training Layer]**
- Stage 1: SFT Training
- DeepSpeed Zero-3 Implementation
- Stage 2: RL Training
- PPO Trainer
- Reward Model (Medical Verifier)
- Value Model

**[Model Deployment Layer]**
- Inference Pipeline
- vLLM Integration
- SGLang Integration

**[Evaluation System]**
- SGLang Server
- Evaluator
- Scoring System

##### Technical Details

- Model Configurations:
- Different Model Sizes (7B-70B)
- Supported Languages (English/Chinese)
- Hardware Requirements

- Integration Points:
- External APIs (GPT-4)
- Deployment Options (vLLM, SGLang)
- Evaluation Tools

##### Additional Notes

- Version information for major components
- Scaling capabilities and resource requirements
- Error handling and feedback loops
- Document the "thinks-before-it-answers" approach in the inference flow

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph DPL["Data Processing Layer"]
        VPC["Verifiable Problem Constructor"]
        CRP["Complex Reasoning Path Searcher"]
        TDP["Training Data Pipeline"]
    end

    subgraph MTL["Model Training Layer"]
        subgraph SFT["Stage 1: SFT Training"]
            SFTI["SFT Implementation"]
            DSZ["DeepSpeed Zero-3"]
        end
        
        subgraph RLT["Stage 2: RL Training"]
            PPOT["PPO Trainer"]
            RMV["Reward Model (Medical Verifier)"]
            VM["Value Model"]
        end
    end

    subgraph MDL["Model Deployment Layer"]
        IP["Inference Pipeline"]
        VLLM["vLLM Integration"]
        SGL["SGLang Integration"]
    end

    subgraph EVS["Evaluation System"]
        SGLS["SGLang Server"]
        EVAL["Evaluator"]
        SCR["Scoring System"]
    end

    %% Data Flow Connections
    VPC --> TDP
    CRP --> TDP
    TDP --> SFTI
    SFTI --> PPOT
    RMV --> PPOT
    VM --> PPOT
    PPOT --> IP
    IP --> VLLM
    IP --> SGL
    SGL --> SGLS
    SGLS --> EVAL
    EVAL --> SCR

    %% Styling
    classDef processing fill:#a8d1ff,stroke:#333,stroke-width:2px
    classDef training fill:#90EE90,stroke:#333,stroke-width:2px
    classDef deployment fill:#FFE5B4,stroke:#333,stroke-width:2px
    classDef evaluation fill:#DDA0DD,stroke:#333,stroke-width:2px

    class VPC,CRP,TDP processing
    class SFTI,DSZ,PPOT,RMV,VM training
    class IP,VLLM,SGL deployment
    class SGLS,EVAL,SCR evaluation

    %% Click Events
    click VPC "https://github.com/FreedomIntelligence/HuatuoGPT-o1/blob/main/construct_verifiable_medical_problems.py"
    click CRP "https://github.com/FreedomIntelligence/HuatuoGPT-o1/blob/main/search_for_complex_reasoning_path.py"
    click TDP "https://github.com/FreedomIntelligence/HuatuoGPT-o1/blob/main/data/demo_data.json"
    click SFTI "https://github.com/FreedomIntelligence/HuatuoGPT-o1/blob/main/SFT_stage1.py"
    click DSZ "https://github.com/FreedomIntelligence/HuatuoGPT-o1/blob/main/configs/deepspeed_zero3.yaml"
    click PPOT "https://github.com/FreedomIntelligence/HuatuoGPT-o1/blob/main/ppo_utils/ppo_trainer_medo1.py"
    click EVAL "https://github.com/FreedomIntelligence/HuatuoGPT-o1/blob/main/evaluation/eval.py"
    click SCR "https://github.com/FreedomIntelligence/HuatuoGPT-o1/blob/main/evaluation/scorer.py"
    click SGLS "https://github.com/FreedomIntelligence/HuatuoGPT-o1/blob/main/evaluation/kill_sglang_server.sh"
```
