How tencent-hunyuan/hunyuan-a13b works
Project type and purpose
• This is an open-source large-language-model (LLM) repository (Hunyuan-A13B) providing:
• Model definition & configuration (Mixture-of-Experts architecture)
• Training pipelines (data prep, fine-tuning, merging LoRA weights)
• Inference pipelines (API server scripts for various backends: TensorRT-LLM, vLLM, SGLang)
• Demo clients, agent tools, benchmarks, and quantized model support
File structure
• Top level:
• models/ – model code (configuration, tokenization, modeling)
• train/ – training configs, scripts, example data, merge utilities, DS-Zero JSON files
• inference/ – scripts to launch API servers (FP8, Int4, Int8, BF16) and client helpers
• examples/ – sample data, training & inference demos, eval scripts
• agent/ – agent-style demo tools (Excel parsing, MCP demo, custom parsers)
• benchmark/ – benchmark scripts and results
• report/ – the technical PDF report
• root files – READMEs, licensing, requirements, submodule references
README for additional insights
• Core LLM (80B total, 13B active parameters) with MoE
• Dual “thinking” modes (fast/slow reasoning)
• Ultra-long context (256K tokens)
• Quantization via AngleSlim: FP8 and GPTQ-Int4
• Supported backends: TensorRT-LLM, vLLM, SGLang
• Deployment via Docker images, OpenAI-compatible API
Final Recommendations
• Be as granular as needed: e.g., separate “Tokenization” from “Model Forward” if desired
• Both horizontal layers (Data→Training→Deployment) and vertical stacks (Model code reused across pipelines)
• With versioned config files (e.g., ds_zero3_offload.json) to highlight custom setups