# thu-pacman/chitu architecture

> Architecture diagram of the GitHub repository thu-pacman/chitu, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thu-pacman/chitu
- Repository: https://github.com/thu-pacman/chitu

## Overview

##### And Outline the Main Components
- Inference Engine Core: Represent the core engine found in files like chitu/chitu_main.py, chitu/backend.py, and chitu/serve.py. This component orchestrates inference operations.
- Model and Configuration Manager: Include the “chitu/config” directory that holds various YAML configuration files for different models and service setups. Also show “chitu/models” (and submodules like model_deepseek_v3.py, model_hf_llama.py) as integral to loading and interfacing with different models.
- GPU Kernel & CUDA Components: Include a component depicting the custom CUDA kernels and C++ integration found in the “csrc” directory to accelerate operations (note the kernel source files and their role in operations like fused MOE and attention).
- Scheduler and Task Management: Represent the scheduler (chitu/scheduler.py) and executor (chitu/executor.py) which handle parallelism (both tensor and pipeline parallelism) and task distribution across GPUs and nodes.
- External Interfaces and Service Layer: Clearly indicate the service endpoints and REST API layer used to receive inference requests (as shown in the “torchrun … chitu serve.port=…” command example). This should also cover the interaction with clients (e.g., via curl or external load-test scripts) and the orchestration logic.
- Benchmarking and Testing: Represent supporting components for performance evaluation—include the “benchmarks” and “test” directories that provide scripts for both single request testing and comprehensive performance benchmarking.

##### Relationships and Interactions
- Parallelism and Distributed Operations: Illustrate how torchrun and multi-node execution work with modules like device_list.py and device_type.py to distribute the workload across multiple GPUs/nodes.
- Auxiliary Components: Include areas for caching (chitu/cache_manager.py) and any accelerator-specific optimizations (like cuda_graph in chitu/cuda_graph.py).

##### Architectural Patterns and Design Principles
- Modular and Extensible Design: Show that Chitu is broken into logically distinct modules (serving, models, quantization, GPU kernels) that can be developed or replaced independently.
- Hybrid Parallelism: Emphasize the separation and combination of tensor parallelism (TP) and pipeline parallelism (PP) as signaled by configurations in the README.
- Abstraction of Hardware-Specific Optimizations: The project abstracts GPU kernel implementations (in C++ and CUDA) away from the high-level Python orchestration.

##### Relevant Technologies, Frameworks, and Libraries
- Python is the primary language; major components are written in Python.
- PyTorch is used as the backbone for model handling and distributed inference (as seen in the torchrun commands).
- CUDA/C++ integration for GPU kernel optimizations, located in the “csrc” directory.
- Docker and CI/CD integrations (Dockerfile, GitHub/GitLab workflows) may also be noted as part of deployment/reproducibility aspects.
- Third-party libraries and submodules (e.g., AutoGPTQ, FlashMLA) are integrated, as seen in the “third_party” directory.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Client Layer
    subgraph "Client Layer"
        Client["Client Request"]:::client
    end

    %% Application Layer
    subgraph "Application Layer"
        subgraph "Task Management"
            Scheduler["Scheduler (scheduler.py)"]:::scheduler
            Executor["Executor (executor.py)"]:::scheduler
        end
        subgraph "Inference Engine Core"
            Main["chitu_main.py"]:::inference
            Backend["backend.py"]:::inference
            Serve["serve.py"]:::inference
        end
    end

    %% Model & Configuration Manager
    subgraph "Model & Configuration Manager"
        Config["Config Manager (chitu/config)"]:::config
        Model["Model Loader (chitu/models)"]:::config
    end

    %% Quantization & Optimization
    subgraph "Quantization & Optimization"
        Quantize["Quantization (chitu/quantize)"]:::quant
        AWQ["AWQ Module (chitu/awq)"]:::quant
        CUDA["CUDA Kernels (csrc)"]:::cuda
    end

    %% Auxiliary
    subgraph "Auxiliary"
        Benchmark["Benchmarking (benchmarks)"]:::aux
        Testing["Testing (test)"]:::aux
    end

    %% Data Flow Connections
    Client -->|"API_request"| Serve
    Serve -->|"forwards_to"| Scheduler
    Scheduler -->|"distributes"| Executor
    Executor -->|"triggers"| Main
    Main -->|"loads"| Config
    Main -->|"loads"| Model
    Config -->|"provides"| Main
    Model -->|"provides"| Main
    Main -->|"applies"| Quantize
    Quantize -->|"integrates"| AWQ
    Quantize -->|"executes"| CUDA
    CUDA -->|"returns"| Main
    Main -->|"sends_result"| Executor
    Executor -->|"passes"| Serve
    Serve -->|"responds"| Client
    Benchmark -->|"tests"| Serve
    Testing -->|"tests"| Serve

    %% Click Events
    click Main "https://github.com/thu-pacman/chitu/blob/public-main/chitu/chitu_main.py"
    click Backend "https://github.com/thu-pacman/chitu/blob/public-main/chitu/backend.py"
    click Serve "https://github.com/thu-pacman/chitu/blob/public-main/chitu/serve.py"
    click Scheduler "https://github.com/thu-pacman/chitu/blob/public-main/chitu/scheduler.py"
    click Executor "https://github.com/thu-pacman/chitu/blob/public-main/chitu/executor.py"
    click Config "https://github.com/thu-pacman/chitu/tree/public-main/chitu/config"
    click Model "https://github.com/thu-pacman/chitu/tree/public-main/chitu/models"
    click Quantize "https://github.com/thu-pacman/chitu/tree/public-main/chitu/quantize"
    click AWQ "https://github.com/thu-pacman/chitu/tree/public-main/chitu/awq"
    click CUDA "https://github.com/thu-pacman/chitu/tree/public-main/csrc"
    click Benchmark "https://github.com/thu-pacman/chitu/tree/public-main/benchmarks"
    click Testing "https://github.com/thu-pacman/chitu/tree/public-main/test"

    %% Styles
    classDef client fill:#f9e79f,stroke:#d4ac0d,stroke-width:2px;
    classDef scheduler fill:#d6eaf8,stroke:#3498db,stroke-width:2px;
    classDef inference fill:#fadbd8,stroke:#e74c3c,stroke-width:2px;
    classDef config fill:#d5f5e3,stroke:#28b463,stroke-width:2px;
    classDef quant fill:#f5cba7,stroke:#e67e22,stroke-width:2px;
    classDef cuda fill:#e8daef,stroke:#8e44ad,stroke-width:2px;
    classDef aux fill:#f0b27a,stroke:#d35400,stroke-width:2px;
```
