# scienceone-ai/deepseek-671b-sft-guide architecture

> Architecture diagram of the GitHub repository scienceone-ai/deepseek-671b-sft-guide, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/scienceone-ai/deepseek-671b-sft-guide
- Repository: https://github.com/scienceone-ai/deepseek-671b-sft-guide

## Overview

##### Project Type & Purpose
- This project is a specialized deep-learning training and deployment framework focused on full-parameter fine-tuning for a massive language model (DeepSeek-V3/R1 671B).
- Its purposes include: preparing and converting training data (with extended reasoning formats), executing distributed training with both data parallel (via DeepSpeed ZeRO) and sequence parallel (SP) strategies, and converting/tracing the final model weights into a Huggingface-compatible format for deployment (using vLLM over a Ray cluster).

##### Main Components to Represent (Each as a module
a. Data Preparation & Preprocessing Module
- Components: “data” folder (e.g., train_example.json sample), dataset conversion logic in code/xtuner/dataset, map functions (alpaca, arxiv, code_alpaca, etc.), and custom/preprocess scripts.
- Flow: Raw training data → Data converters/formatters → Training-ready tensor formats.

b. Model & Configuration Layer
- Components: "model" directory (DeepSeek-V3/R1 modeling files, custom modifications from DeepSeek-V2/V3 papers), configuration files under “code/xtuner/configs” for various run modes (e.g., zero/offload, parallel strategies), and modified “xtuner” core code.
- Flow: Configurations, custom modeling modules, datasets, and parameter settings feed into model instantiation.

c. Training Engine & Distributed Training Infrastructure
- Components: “code/xtuner/engine” (runner, hooks, parallel setups), integration with DeepSpeed (ZeRO configurations in code/xtuner/configs/deepspeed), and support for both data parallelism and sequence parallelism.
- Connection: Training scripts (e.g., sft_deepseek.py and corresponding shell scripts) activate the engine which uses the configured parallel strategies. Include scheduler integration (slurm, pdsh) and environment variable setups.
- Additional Details: Model modules in “code/xtuner/model” define the training logic; engine hooks (e.g., dataset info, evaluation, checkpointing) manage runtime behavior and logging.

d. Checkpointing & Intermediate Storage
- Components: Intermediate model weights in “pth” format (noted in training – each checkpoint ~7.4TB), conversion scripts under “code/scripts/convert_pth_to_hf.sh”.
- Flow: Conversion of raw pth weights to Huggingface model format to support subsequent deployment.

e. Deployment & Inference Layer
- Components: vLLM deployment scripts, Ray cluster orchestration (ray start commands in vllm_deploy_slurm.sh), and integration with Huggingface Transformers.
- Flow: Final converted model → vLLM serving (using ray cluster with tensor-/pipeline-parallel size settings) → Client APIs (curl example).

##### Relationships and Data Flow
which then feeds training-ready inputs into the Training Engine.
- The Training Engine interacts with model instantiation (from the Model & Configuration layer) and uses distributed parallel strategies.
- After model training, checkpoint outputs pass to the Checkpoint Conversion module.
- The final model is deployed by the Inference engine (vLLM deployment), and client requests flow from external API calls into the serving system.
- That underlying libraries (DeepSpeed, Huggingface transformers, xtuner framework, and Ray) form the technology backbone in various layers.

##### Architectural Design Patterns & Technologies
- The architecture follows a layered design with clear separation between data preprocessing, training (with distributed strategies), and deployment.
- It employs modularity (each “config” folder for various parallel and fine-tuning setups) and plug-in patterns (for dataset mapping functions, training hooks, and model converters).
- The design also leverages distributed computing patterns (data parallelism via DeepSpeed ZeRO offload and sequence parallelism) along with inter-node communication using SLURM/PDSH for orchestration.
- Technologies and frameworks: Python v3.10, DeepSpeed, Ray, vLLM, Huggingface Transformers, and xtuner.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Preparation & Pre-Processing Subgraph
    subgraph "Data Preparation & Pre-Processing"
        DP1["Data Files"]:::data
        DP2["Dataset Conversion & Mapping"]:::data
        DP3["Custom Preprocessing Scripts"]:::data
        DP1 -->|"converts"| DP2
        DP2 -->|"formats"| DP3
    end

    %% Model & Configuration Subgraph
    subgraph "Model & Configuration"
        MC1["Model Files & Variants"]:::model
        MC2["Configurations & xtuner Core"]:::model
        MC1 -->|"config feeds"| MC2
    end

    %% Distributed Training Engine Subgraph
    subgraph "Distributed Training Engine"
        TE4a["Training Script: sft_deepseek.py"]:::training
        TE4b["Training Script: sft_deepseek.sh"]:::training
        TE4c["Training Script: sft_deepseek_pdsh.sh"]:::training
        TE1["Engine Core & Runner"]:::training
        TE2["Engine Hooks"]:::training
        TE3["Distributed Strategies (DP via DeepSpeed ZeRO & SP)"]:::training
        
        TE4a -->|"initiates"| TE1
        TE4b -->|"initiates"| TE1
        TE4c -->|"initiates"| TE1
        TE1 -->|"utilizes"| TE2
        TE1 -->|"employs"| TE3
    end

    %% Checkpoint Storage & Weight Conversion Subgraph
    subgraph "Checkpoint Storage & Weight Conversion"
        CP1["Checkpoint Conversion Script"]:::checkpoint
    end

    %% Inference & Deployment Subgraph
    subgraph "Inference & Deployment"
        ID1["vLLM & Ray Deployment"]:::deploy
    end

    %% External Client API
    E1["Client API Requests"]:::external

    %% Flow Connections between major modules
    DP3 -->|"preprocessed data"| TE1
    MC2 -->|"model parameters"| TE1
    TE1 -->|"trained model"| CP1
    CP1 -->|"converted model"| ID1
    E1 -->|"inference calls"| ID1

    %% Click Events for Data Preparation & Pre-Processing
    click DP1 "https://github.com/scienceone-ai/deepseek-671b-sft-guide/blob/main/data/train_example.json"
    click DP2 "https://github.com/scienceone-ai/deepseek-671b-sft-guide/tree/main/code/xtuner/dataset"
    click DP3 "https://github.com/scienceone-ai/deepseek-671b-sft-guide/tree/main/code/xtuner/tools/data_preprocess"

    %% Click Events for Model & Configuration
    click MC1 "https://github.com/scienceone-ai/deepseek-671b-sft-guide/tree/main/model/DeepSeek-V3-BF16"
    click MC2 "https://github.com/scienceone-ai/deepseek-671b-sft-guide/tree/main/code/xtuner/configs"

    %% Click Events for Training Engine & Distributed Training Infrastructure
    click TE1 "https://github.com/scienceone-ai/deepseek-671b-sft-guide/tree/main/code/xtuner/engine"
    click TE2 "https://github.com/scienceone-ai/deepseek-671b-sft-guide/tree/main/code/xtuner/engine/hooks"
    click TE4a "https://github.com/scienceone-ai/deepseek-671b-sft-guide/blob/main/code/scripts/sft_deepseek.py"
    click TE4b "https://github.com/scienceone-ai/deepseek-671b-sft-guide/blob/main/code/scripts/sft_deepseek.sh"
    click TE4c "https://github.com/scienceone-ai/deepseek-671b-sft-guide/blob/main/code/scripts/sft_deepseek_pdsh.sh"

    %% Click Event for Checkpointing & Intermediate Storage
    click CP1 "https://github.com/scienceone-ai/deepseek-671b-sft-guide/blob/main/code/scripts/convert_pth_to_hf.sh"

    %% Click Event for Deployment & Inference
    click ID1 "https://github.com/scienceone-ai/deepseek-671b-sft-guide/blob/main/code/scripts/vllm_deploy_slurm.sh"

    %% Styles
    classDef data fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef model fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef training fill:#FFA07A,stroke:#000,stroke-width:2px;
    classDef checkpoint fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef deploy fill:#FFB6C1,stroke:#000,stroke-width:2px;
    classDef external fill:#E0E0E0,stroke:#000,stroke-width:2px;
```
