How tencent-hunyuan/tencent-hunyuan-large works
Project Type and Purpose
• This is an open-source, full-pipeline Large Language Model (LLM) project (“Hunyuan-Large”) providing:
• Model definitions and tokenizer code under models/
• Training scripts and DeepSpeed configurations under train/
• Inference scripts and service definitions under inference/
• Example usages and data under examples/
• Benchmark scripts under benchmark/
• Purpose: enable researchers to fine-tune and deploy a 389B-parameter MoE Transformer model using Hugging Face, DeepSpeed and vLLM backends.
File-Structure Analysis
• Top-level directories:
• models/ – Python modules: configuration_hunyuan.py, modeling_hunyuan.py, tokenizer code, generation config
• train/ – training entrypoints: train.py, shell script train.sh, DeepSpeed JSON configs, LoRA merge scripts
• inference/ – inference entrypoints: SSH helper, Docker startup scripts, vLLM server scripts, OpenAPI client
• examples/ – sample data, usage scripts for training (train_demo.sh), evaluation, build environments
• benchmark/ – evaluation harness
• Config and dependency files: requirements.txt, models/generation_config.json, tokenizer config, DeepSpeed JSONs.
README Insights
• Architecture splits into three logical layers:
a. Model layer: MoE Transformer + custom tokenizer
b. Training layer: Hugging Face Trainer + DeepSpeed Zero3 + optional LoRA
c. Inference layer: vLLM‐based service (future TensorRT‐LLM), Ray‐backed distributed RPC
• Multi‐node orchestration via passwordless SSH (train) and Ray cluster (inference).
a. Main Components
1. Data Store – JSONL files & example data (examples/data/*.jsonl, user‐provided datasets)
2. Pre-processing Module – examples/compute_acc.py, data formatting in train/train.py
3. Training Controller – train.sh launches train.py with DeepSpeed configs
4. DeepSpeed Cluster – multi-node Zero3 workers coordinated over SSH
5. Model Definition – models/modeling_hunyuan.py, configuration_hunyuan.py, tokenizer modules
6. Checkpoint Storage – local or shared filesystem for model weights, LoRA adapters
7. LoRA Merge Utility – merge_lora_weight.py / merge_lora_weight.sh
8. Docker Image – hunyuaninfer/hunyuan-large:infer-open-source for inference
9. Ray Cluster – head and worker nodes started via ray start
10. vLLM Inference Service – run_server.sh (+ run_server_int8.sh / run_server_fp8.sh)
11. API Gateway / Client – openapi.sh, OpenAPI HTTP endpoint for external requests
b. Relationships & Data Flows
• Data Store → Preprocessing → Training Controller
• Training Controller ↔ SSH → DeepSpeed Cluster
• DeepSpeed Cluster ↔ Model Definition → Checkpoint Storage
• Checkpoint Storage → LoRA Merge Utility → Final Weights
• Checkpoint Storage → Docker Image → Ray Cluster
• Ray Cluster ← vLLM Service ← API Gateway ← HTTP Client
• vLLM Service → Model Definition & Tokenizer
c. Architectural Patterns
• Master-Worker for both training (SSH orchestrator + DeepSpeed) and inference (Ray head + Ray workers)
• Modularization: clear separation of model code, training pipeline, inference pipeline
• Extensibility points: plug in different DeepSpeed .json configs, switch quantization modes, swap inference backend (vLLM vs TRT-LLM)
d. Key Technologies
• Python + Hugging Face Transformers + custom modeling_hunyuan.py
• DeepSpeed Zero3 & hf-deepspeed trainer
• LoRA adapters for parameter‐efficient fine-tuning
• vLLM backend, Ray distributed runtime
• Docker containers for consistent inference environments
Key Takeaway
• Break the project into logical subsystems: Data → Training → Checkpointing → Inference
• Orchestration layers (SSH vs Ray), technology stacks, and extension points for quantization, benchmarking, and deployment