How thu-pacman/chitu works
And Outline the Main Components
• Inference Engine Core: Represent the core engine found in files like chitu/chitu_main.py, chitu/backend.py, and chitu/serve.py. This component orchestrates inference operations.
• Model and Configuration Manager: Include the “chitu/config” directory that holds various YAML configuration files for different models and service setups. Also show “chitu/models” (and submodules like model_deepseek_v3.py, model_hf_llama.py) as integral to loading and interfacing with different models.
• GPU Kernel & CUDA Components: Include a component depicting the custom CUDA kernels and C++ integration found in the “csrc” directory to accelerate operations (note the kernel source files and their role in operations like fused MOE and attention).
• Scheduler and Task Management: Represent the scheduler (chitu/scheduler.py) and executor (chitu/executor.py) which handle parallelism (both tensor and pipeline parallelism) and task distribution across GPUs and nodes.
• External Interfaces and Service Layer: Clearly indicate the service endpoints and REST API layer used to receive inference requests (as shown in the “torchrun … chitu serve.port=…” command example). This should also cover the interaction with clients (e.g., via curl or external load-test scripts) and the orchestration logic.
• Benchmarking and Testing: Represent supporting components for performance evaluation—include the “benchmarks” and “test” directories that provide scripts for both single request testing and comprehensive performance benchmarking.
Relationships and Interactions
• Parallelism and Distributed Operations: Illustrate how torchrun and multi-node execution work with modules like device_list.py and device_type.py to distribute the workload across multiple GPUs/nodes.
• Auxiliary Components: Include areas for caching (chitu/cache_manager.py) and any accelerator-specific optimizations (like cuda_graph in chitu/cuda_graph.py).
Architectural Patterns and Design Principles
• Modular and Extensible Design: Show that Chitu is broken into logically distinct modules (serving, models, quantization, GPU kernels) that can be developed or replaced independently.
• Hybrid Parallelism: Emphasize the separation and combination of tensor parallelism (TP) and pipeline parallelism (PP) as signaled by configurations in the README.
• Abstraction of Hardware-Specific Optimizations: The project abstracts GPU kernel implementations (in C++ and CUDA) away from the high-level Python orchestration.
Relevant Technologies, Frameworks, and Libraries
• Python is the primary language; major components are written in Python.
• PyTorch is used as the backbone for model handling and distributed inference (as seen in the torchrun commands).
• CUDA/C++ integration for GPU kernel optimizations, located in the “csrc” directory.
• Docker and CI/CD integrations (Dockerfile, GitHub/GitLab workflows) may also be noted as part of deployment/reproducibility aspects.
• Third-party libraries and submodules (e.g., AutoGPTQ, FlashMLA) are integrated, as seen in the “third_party” directory.