# tencent-hunyuan/hunyuan-turbos3 architecture

> Architecture diagram of the GitHub repository tencent-hunyuan/hunyuan-turbos3, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencent-hunyuan/hunyuan-turbos3
- Repository: https://github.com/tencent-hunyuan/hunyuan-turbos3

## Overview

##### Project type and purpose
- This is not a standard application codebase but a research-grade large language model (LLM) project.
- Hunyuan-TurboS is a 56B-activated parameter hybrid Transformer-Mamba MoE model with an adaptive chain-of-thought mechanism.
- The README points to a technical report (PDF) for full details and describes both model architecture and multi-stage training pipeline.

##### File structure
- Top level contains only README.md and the technical report PDF – no code.

##### README for additional insights
- Key architecture: Mamba2 blocks for long-sequence efficiency, standard Transformer Attention layers, Mixture-of-Experts (MoE) feed-forward networks with grouped-query attention.
- Adaptive long-short chain-of-thought controller that switches inference modes.
- Training stages: Pre-training on 16T tokens, Supervised Fine-Tuning (SFT), Adaptive CoT Fusion, Multi-round Deliberation Learning, and two-stage RL.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Documentation
    Docs["Project Docs"]:::external
    subgraph "Documentation"
        README["README.md"]:::external
        Report["Hunyuan-Turbos_Report.pdf"]:::external
    end

    %% Model Architecture View
    subgraph "Model Architecture View"
        direction TB
        Input["Input Text\n(256K context)"]:::external
        Tokenizer["Tokenizer"]:::core
        Embedding["Embedding Layer"]:::core
        Controller["Adaptive CoT Controller"]:::control
        subgraph "Fast Path (Mamba2)" 
            direction TB
            Mamba1["Mamba2 Block"]:::core
            Mamba2["Mamba2 Block"]:::core
        end
        subgraph "Deep Path (Alternating Blocks)"
            direction TB
            A1["Mamba2 Block"]:::core
            A2["Transformer Attention + MoE FFN"]:::core
            A3["Mamba2 Block"]:::core
            A4["Transformer Attention + MoE FFN"]:::core
        end
        Merge["Merge Paths"]:::control
        KVCache["KV Cache & Memory Mgmt"]:::core
        Output["Output Head / Logits Projection"]:::core
        Decode["Text Decode"]:::external

        Input --> Tokenizer --> Embedding --> Controller
        Controller -->|"simple query"| Mamba1
        Controller -->|"complex query"| A1
        Mamba1 --> Mamba2 --> Merge
        A1 --> A2 --> A3 --> A4 --> Merge
        Merge --> KVCache --> Output --> Decode
    end

    %% Training Pipeline View
    subgraph "Training Pipeline View"
        direction TB
        RawData["Raw Data\n(16T tokens)"]:::pipeline
        PreProc["Pre-processor"]:::pipeline
        PreTrain["Pre-training Engine"]:::pipeline
        SFT["SFT Module\n(3M instructions)"]:::pipeline
        CoTFusion["Adaptive CoT Fusion Module"]:::pipeline
        Delib["Multi-round Deliberation Trainer"]:::pipeline
        RL1["RL Phase I Module"]:::pipeline
        RL2["RL Phase II Module"]:::pipeline
        Artifact["Final Model Artifact"]:::pipeline
        Deploy["Deployment / Inference Pod"]:::pipeline

        RawData --> PreProc --> PreTrain --> SFT --> CoTFusion --> Delib --> RL1 --> RL2 --> Artifact --> Deploy
    end

    %% Click Events
    click README "https://github.com/tencent-hunyuan/hunyuan-turbos3/blob/main/README.md"
    click Report "https://github.com/tencent-hunyuan/hunyuan-turbos3/blob/main/Hunyuan-Turbos_Report.pdf"

    %% Styles
    classDef core fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef control fill:#90EE90,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef pipeline fill:#FFA500,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px,stroke-dasharray: 2 2
```
