# tencent-hunyuan/tencent-hunyuan-large architecture

> Architecture diagram of the GitHub repository tencent-hunyuan/tencent-hunyuan-large, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencent-hunyuan/tencent-hunyuan-large
- Repository: https://github.com/tencent-hunyuan/tencent-hunyuan-large

## Overview

##### Project Type and Purpose
- This is an open-source, full-pipeline Large Language Model (LLM) project (“Hunyuan-Large”) providing:
- Model definitions and tokenizer code under `models/`
- Training scripts and DeepSpeed configurations under `train/`
- Inference scripts and service definitions under `inference/`
- Example usages and data under `examples/`
- Benchmark scripts under `benchmark/`
- Purpose: enable researchers to fine-tune and deploy a 389B-parameter MoE Transformer model using Hugging Face, DeepSpeed and vLLM backends.

##### File-Structure Analysis
- Top-level directories:
- models/ – Python modules: `configuration_hunyuan.py`, `modeling_hunyuan.py`, tokenizer code, generation config
- train/ – training entrypoints: `train.py`, shell script `train.sh`, DeepSpeed JSON configs, LoRA merge scripts
- inference/ – inference entrypoints: SSH helper, Docker startup scripts, vLLM server scripts, OpenAPI client
- examples/ – sample data, usage scripts for training (`train_demo.sh`), evaluation, build environments
- benchmark/ – evaluation harness
- Config and dependency files: `requirements.txt`, `models/generation_config.json`, tokenizer config, DeepSpeed JSONs.

##### README Insights
- Architecture splits into three logical layers:
a. Model layer: MoE Transformer + custom tokenizer
b. Training layer: Hugging Face Trainer + DeepSpeed Zero3 + optional LoRA
c. Inference layer: vLLM‐based service (future TensorRT‐LLM), Ray‐backed distributed RPC
- Multi‐node orchestration via passwordless SSH (train) and Ray cluster (inference).

a. Main Components
1. Data Store – JSONL files & example data (`examples/data/*.jsonl`, user‐provided datasets)
2. Pre-processing Module – `examples/compute_acc.py`, data formatting in `train/train.py`
3. Training Controller – `train.sh` launches `train.py` with DeepSpeed configs
4. DeepSpeed Cluster – multi-node Zero3 workers coordinated over SSH
5. Model Definition – `models/modeling_hunyuan.py`, `configuration_hunyuan.py`, tokenizer modules
6. Checkpoint Storage – local or shared filesystem for model weights, LoRA adapters
7. LoRA Merge Utility – `merge_lora_weight.py` / `merge_lora_weight.sh`
8. Docker Image – `hunyuaninfer/hunyuan-large:infer-open-source` for inference
9. Ray Cluster – head and worker nodes started via `ray start`
10. vLLM Inference Service – `run_server.sh` (+ `run_server_int8.sh` / `run_server_fp8.sh`)
11. API Gateway / Client – `openapi.sh`, OpenAPI HTTP endpoint for external requests
b. Relationships & Data Flows
- Data Store → Preprocessing → Training Controller
- Training Controller ↔ SSH → DeepSpeed Cluster
- DeepSpeed Cluster ↔ Model Definition → Checkpoint Storage
- Checkpoint Storage → LoRA Merge Utility → Final Weights
- Checkpoint Storage → Docker Image → Ray Cluster
- Ray Cluster ← vLLM Service ← API Gateway ← HTTP Client
- vLLM Service → Model Definition & Tokenizer
c. Architectural Patterns
- Master-Worker for both training (SSH orchestrator + DeepSpeed) and inference (Ray head + Ray workers)
- Modularization: clear separation of model code, training pipeline, inference pipeline
- Extensibility points: plug in different DeepSpeed `.json` configs, switch quantization modes, swap inference backend (vLLM vs TRT-LLM)
d. Key Technologies
- Python + Hugging Face Transformers + custom `modeling_hunyuan.py`
- DeepSpeed Zero3 & hf-deepspeed trainer
- LoRA adapters for parameter‐efficient fine-tuning
- vLLM backend, Ray distributed runtime
- Docker containers for consistent inference environments

##### Key Takeaway
- Break the project into logical subsystems: Data → Training → Checkpointing → Inference
- Orchestration layers (SSH vs Ray), technology stacks, and extension points for quantization, benchmarking, and deployment

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Store
    subgraph "Data Store"
        direction TB
        DS1["car_train.jsonl"]:::storage
        DS2["car_test.json"]:::storage
        DS3["example_data.jsonl"]:::storage
        DS4["train/example_data.jsonl"]:::storage
    end

    %% Pre-processing
    subgraph "Pre-processing Module"
        direction TB
        ComputeAcc["compute_acc.py"]:::compute
        DataFormatter["train.py (data formatting)"]:::compute
    end

    %% Training Controller
    subgraph "Training Controller"
        direction TB
        TrainShell["train.sh"]:::compute
        TrainLauncher["train.py (entrypoint)"]:::compute
    end

    %% DeepSpeed Cluster Configs
    subgraph "DeepSpeed Cluster"
        direction TB
        SSHOrch["SSH Orchestration"]:::orchestration
        DSCluster["Zero3 Multi-node Workers"]:::compute
        subgraph "Configs"
            direction TB
            DS1cfg["ds_zero2_no_offload.json"]:::compute
            DS2cfg["ds_zero3_no_offload.json"]:::compute
            DS3cfg["ds_zero3_offload.json"]:::compute
            DS4cfg["ds_zero3_offload_no_auto.json"]:::compute
        end
    end

    %% Model Definition
    subgraph "Model Definition"
        direction TB
        ModCode["modeling_hunyuan.py"]:::compute
        ModConfig["configuration_hunyuan.py"]:::compute
        TokenLib["tokenization_hy.py"]:::compute
        Tiktoken["hy.tiktoken"]:::compute
        TokenJson["tokenizer_config.json"]:::compute
        GenConfig["generation_config.json"]:::compute
    end

    %% Checkpoint Storage
    Checkpoints["Checkpoint Storage"]:::storage

    %% LoRA Merge
    LoRAMerge["LoRA Merge Utility"]:::compute

    %% Inference Cluster
    subgraph "Inference Cluster"
        direction TB
        SSHHelper["login_ssh.py"]:::orchestration
        DockerImage["Docker Image"]:::compute
        RayCluster["Ray Cluster"]:::orchestration
    end

    %% vLLM Service
    subgraph "vLLM Inference Service"
        direction TB
        RunFP["run_server_fp8.sh"]:::compute
        RunInt8["run_server_int8.sh"]:::compute
        RunStd["run_server.sh"]:::compute
        VLLMDir["vllm/"]:::compute
    end

    %% API Gateway / Client
    subgraph "API Gateway / Client"
        direction TB
        HTTPClient["HTTP Client"]:::external
        APIGateway["openapi.sh"]:::external
    end

    %% Examples & Benchmark
    subgraph "Examples & Benchmark"
        direction TB
        DemoTrain["train_demo.sh"]:::compute
        EvalDemo["eval_demo_vllm.py"]:::compute
        BuildEnv["build_env.sh"]:::compute
        Benchmark["benchmark/"]:::compute
    end

    %% Connections
    DS1 --> ComputeAcc
    DS2 --> ComputeAcc
    DS3 --> ComputeAcc
    DS4 --> ComputeAcc

    ComputeAcc --> DataFormatter
    DataFormatter --> TrainShell
    DataFormatter --> TrainLauncher

    TrainShell --> SSHOrch
    TrainLauncher --> SSHOrch
    SSHOrch --> DSCluster
    DS1cfg --> DSCluster
    DS2cfg --> DSCluster
    DS3cfg --> DSCluster
    DS4cfg --> DSCluster

    ModCode --> DSCluster
    ModConfig --> DSCluster
    TokenLib --> DSCluster

    DSCluster --> Checkpoints

    Checkpoints --> LoRAMerge
    Checkpoints --> DockerImage

    DockerImage --> RayCluster
    RayCluster --> RunStd
    RayCluster --> RunInt8
    RayCluster --> RunFP
    RayCluster --> VLLMDir

    HTTPClient --> APIGateway
    APIGateway --> RunStd
    APIGateway --> RunInt8
    APIGateway --> RunFP

    RunStd --> ModCode
    RunStd --> TokenLib

    %% Example & Benchmark flows (optional)
    DS1 --> DemoTrain
    DS1 --> EvalDemo
    DemoTrain --> Benchmark

    %% Click Events
    click DS1 "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/examples/data/car_train.jsonl"
    click DS2 "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/examples/data/car_test.json"
    click DS3 "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/examples/data/example_data.jsonl"
    click DS4 "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/train/example_data.jsonl"
    click ComputeAcc "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/examples/compute_acc.py"
    click DataFormatter "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/train/train.py"
    click TrainShell "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/train/train.sh"
    click TrainLauncher "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/train/train.py"
    click DS1cfg "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/train/ds_zero2_no_offload.json"
    click DS2cfg "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/train/ds_zero3_no_offload.json"
    click DS3cfg "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/train/ds_zero3_offload.json"
    click DS4cfg "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/train/ds_zero3_offload_no_auto.json"
    click ModCode "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/models/modeling_hunyuan.py"
    click ModConfig "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/models/configuration_hunyuan.py"
    click TokenLib "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/models/tokenization_hy.py"
    click Tiktoken "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/models/hy.tiktoken"
    click TokenJson "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/models/tokenizer_config.json"
    click GenConfig "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/models/generation_config.json"
    click LoRAMerge "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/train/merge_lora_weight.py"
    click LoRAMerge "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/train/merge_lora_weight.sh"
    click SSHHelper "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/inference/login_ssh.py"
    click DockerImage "https://github.com/tencent-hunyuan/tencent-hunyuan-large/tree/main/hunyuaninfer/hunyuan-large:infer-open-source"
    click RunStd "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/inference/run_server.sh"
    click RunInt8 "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/inference/run_server_int8.sh"
    click RunFP "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/inference/run_server_fp8.sh"
    click VLLMDir "https://github.com/tencent-hunyuan/tencent-hunyuan-large/tree/main/inference/vllm/"
    click APIGateway "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/inference/openapi.sh"
    click DemoTrain "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/examples/train_demo.sh"
    click EvalDemo "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/examples/eval_demo_vllm.py"
    click BuildEnv "https://github.com/tencent-hunyuan/tencent-hunyuan-large/blob/main/examples/build_env.sh"
    click Benchmark "https://github.com/tencent-hunyuan/tencent-hunyuan-large/tree/main/benchmark/"

    %% Styles
    classDef compute fill:#add8e6,stroke:#000,stroke-width:1px
    classDef storage fill:#90ee90,stroke:#000,stroke-width:1px
    classDef orchestration fill:#ffa500,stroke:#000,stroke-width:1px
    classDef external fill:#d3d3d3,stroke:#000,stroke-width:1px
```
