# tencent-hunyuan/hunyuan-turbos architecture

> Architecture diagram of the GitHub repository tencent-hunyuan/hunyuan-turbos, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencent-hunyuan/hunyuan-turbos
- Repository: https://github.com/tencent-hunyuan/hunyuan-turbos

## Overview

##### Project Type and Purpose
- This repository is a research‐oriented model specification and results package for “Hunyuan-TurboS,” a 56B‐activated parameter hybrid Transformer-Mamba Mixture-of-Experts (MoE) large language model.
- It contains: a high-level README, a PDF technical report, and figure assets. There is no application code here—rather it documents the model architecture, training recipe, and evaluation outcomes.

##### Main Components
a. Model Architecture Block
- Input Embedding Layer
- 128 alternating layers of:
– Mamba2 sublayer (linear complexity long-sequence processor)
– Grouped-Query Multi-Head Attention
– Feed-Forward Network (MoE implementation)
- Adaptive Chain-of-Thought (CoT) Router (dynamic switch between “fast path” and “deep path”)
- Output Projection / Language Modeling Head
b. Training Pipeline
- Pre-training on 16T tokens (data ingestion, tokenization)
- Supervised Fine-Tuning (3M instruction examples)
- Adaptive Long-Short CoT Fusion stage
- Multi-Round Deliberation Learning loop
- Two-Stage Reinforcement Learning (RLHF targeting STEM & general instructions)
c. Evaluation & Deployment
- Automated Benchmarks (math, code, reasoning, languages)
- Human Evaluation (LMSYS Chatbot Arena)
- Inference serving with 256K context support

##### Relationships and Data Flow
- Training pipeline flows: Pre-train → SFT → CoT Fusion → Deliberation → RL; each stage refines model weights.

##### Architectural Patterns and Design Principles
- Mixture-of-Experts: distinct MoE routers inside FFN sublayers—indicate expert shards and gating.
- Linear-complexity vs quadratic block interleave (Mamba2 vs attention).
- Adaptive routing for chain-of-thought: illustrate decision node that chooses shallow or deep compute path.
- Large-context handling: separate highlight of 256K context KV cache management.

##### Essentials to Capture
- Precise layer sequencing and hybridization pattern.
- Adaptive CoT mechanism and how it branches compute.
- Distinct MoE expert modules inside FFNs.
- Training stages as chronological or layered processes.
- Data flow direction, technology terms (Group-Query Attention, Mamba2, MoE).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Documentation
    subgraph "Documentation"
        direction TB
        README["High-Level README"]:::doc
        REPORT["PDF Technical Report"]:::doc
        ASSETS["Figure Assets"]:::doc
        EMBFIG["Embedded Figures in Report"]:::doc
    end

    %% Model Architecture
    subgraph "Model Architecture"
        direction TB
        Input["Input Tokens"]:::pipeline
        Embedding["Input Embedding Layer\n(56B active parameters)"]:::core
        subgraph "Transformer Layer Block (repeated 128×)"
            direction TB
            Mamba2["Mamba2 Sublayer\n(linear complexity)"]:::core
            Attention["Grouped-Query Multi-Head Attention\n(quadratic)"]:::core
            FFN["Feed-Forward Network\n(MoE experts + gating)"]:::core
        end
        Router["Adaptive Chain-of-Thought Router\n(fast vs deep path)"]:::router
        Head["Output Projection / LM Head"]:::core
        Output["Output Tokens"]:::pipeline

        Input -->|"tokens"| Embedding
        Embedding -->|"embedded tokens"| Mamba2
        Mamba2 -->|"features"| Attention
        Attention -->|"attended features"| FFN
        FFN -->|"layer output"| Mamba2
        FFN --> Router
        Router -->|"routed features"| Head
        Head --> Output
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        direction TB
        Pre["Pre-training\n16T tokens"]:::pipeline
        SFT["Supervised Fine-Tuning\n3M instruction examples"]:::pipeline
        Fusion["Adaptive Long-Short CoT Fusion"]:::pipeline
        Delib["Multi-Round Deliberation Learning"]:::pipeline
        RL["Two-Stage RLHF\n(STEM & general)"]:::pipeline

        Pre --> SFT --> Fusion --> Delib --> RL
    end

    %% Evaluation & Deployment
    subgraph "Evaluation & Deployment"
        direction TB
        Trained["Trained Model Weights\n(56B active)"]:::core
        Serve["Inference Serving\n256K context"]:::eval
        Bench["Automated Benchmarks"]:::eval
        Human["Human Evaluation\n(LMSYS Chatbot Arena)"]:::eval

        RL --> Trained
        Trained --> Serve
        Serve --> Bench
        Serve --> Human
    end

    %% Click Events
    click README "https://github.com/tencent-hunyuan/hunyuan-turbos/blob/main/README.md"
    click REPORT "https://github.com/tencent-hunyuan/hunyuan-turbos/blob/main/Hunyuan-Turbos_Report.pdf"
    click ASSETS "https://github.com/tencent-hunyuan/hunyuan-turbos/tree/main/Figures/"
    click EMBFIG "https://github.com/tencent-hunyuan/hunyuan-turbos/blob/main/Figures/Hunyuan_TurboS_Report.pdf"

    %% Styles
    classDef core fill:#AED6F1,stroke:#2E86C1,color:#1B4F72;
    classDef router fill:#ABEBC6,stroke:#28B463,color:#1D8348;
    classDef pipeline fill:#F9E79F,stroke:#D4AC0D,color:#7D6608;
    classDef eval fill:#D5DBDB,stroke:#626567,color:#1C2833;
    classDef doc fill:#E8DAEF,stroke:#9B59B6,color:#4A235A;
```
