# ztxz16/fastllm architecture

> Architecture diagram of the GitHub repository ztxz16/fastllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ztxz16/fastllm
- Repository: https://github.com/ztxz16/fastllm

## Overview

Based on your review of the project’s file tree and README, the project is a high-performance large language model (LLM) inference engine implemented in C++ with a “no-backend-dependency” design (apart from CUDA for GPU support). It is built to support mixed inference modes—including CPU, CUDA, NUMA, MOE (mixture of experts), and multi-GPU setups—and also provides Python bindings (pyfastllm) and multiple example applications (CLI demo, web UI, API server, and even mobile/Android examples).

##### Main Components
- Core Inference Engine: Represent both the core C++ library (found in the src and include directories, including modules such as fastllm.cpp, executor.cpp, graph.cpp, and model-specific implementations) and its support for various computation devices (CPU, CUDA, multi-cuda, numa, tfacc, tops, etc. in the include/devices and src/devices folders).
- Device Abstraction Layers: Show separate modules for each type of compute device (CPU, CUDA, NUMA, etc.) that the engine employs for hardware-specific optimizations.
- Model Handling and Conversion: Include components related to model loading, conversion, and quantization. This covers the model directories (both C++ and Python wrappers) as well as the export tool (ftllm.export) and helper scripts under tools/scripts.
- API/Interface Layers: Illustrate the multiple ways the inference engine is exposed:
- Command Line Interface (CLI) – evidenced by command demos (ftllm run, ftllm server, ftllm webui, etc.).
- Web UI and OpenAI-style API server – found in tools/fastllm_pytools and example/openai_server.
- Python Bindings – represented by the “pyfastllm” directory which provides wrappers, additional functions, and custom model structure support.
- Example Applications: Show that there are separate sample projects including Android, Win32Demo, and others (like apiserver, benchmark, and a web UI demo) that demonstrate how to deploy or integrate the core library.

##### Relationships and Interactions
- The core engine (with its device layers) is at the heart of the system. It directly accepts inference requests.
- On top of the core, API and interface layers (CLI, server, web UI, Python scripts) issue commands and fetch inference results.
- Model conversion/export and utility tools interact with the core engine’s input and output (model weights processing, quantization, etc.), and these can be mapped as pre-deployment stages.
- The examples (Android demo, desktop demos) are client-side applications that use the inference engine directly or through wrappers.
- Integration with external services (e.g., model download via Hugging Face endpoints, caching mechanisms, etc.) can be represented as external data flows interfacing with the engine.

##### Architectural Patterns and Design Principles
- Modular and Layered Architecture: Core components (inference, device abstraction), middleware (conversion tools, command interfaces), and front-end applications (CLI, web UI, Android demo) are clearly separated.
- Extensibility: The design supports adding new computation devices (e.g., new GPU models or NUMA configurations) due to its decoupled device operations.
- Cross-platform and Deployment Flexibility: With C++ core and minimal runtime dependencies, the engine is designed to run on diverse OSes (Linux, Windows, Android).
- Interoperability: Python bindings and open API endpoints make it easier to integrate with other systems or customize model structures.

##### Relevant Technologies and Components
- C++ with CMake: The core is built using CMake (see CMakeLists.txt and install.sh scripts).
- CUDA: Critical for GPU-acceleration, as indicated by GPU-specific modules and compile flags.
- Python Bindings: The “pyfastllm” subproject gives Python-callable APIs and conversion scripts.
- Web Technologies: HTML, CSS, JS files in example/webui for frontend demonstrations.
- External Tools: Docker and docker-compose files suggest containerization support.
- Version Control and CI/CD: .github/workflows and other configuration files highlight integration pipelines.

##### Detailed Component Breakdown
- Branching from this, create separate nodes for “Device Modules” (subnodes for CPU, CUDA, NUMA, tfacc, etc.) showing their low-level integration with the core.
- Overlay interfaces to illustrate how the core is accessed:
- CLI Layer – maps to command-line tools.
- API Server/Web UI – shows RESTful API endpoints.
- Python Interface – displays modules under pyfastllm.
- For deployment, showcase containerization (Dockerfile, docker-compose), CI/CD integration (.github/workflows), and additional build scripts.

This approach is tailored to a full-stack system design (with a heavy backend/inference engine) and takes into account the core functionality, extensibility, and deployment considerations specific to this LLM inference project.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Layer 1: Core Engine"
        CE["FastLLM Core Library"]:::core
        subgraph "Device Abstraction Layers"
            CPU["CPU Module"]:::device
            CUDA["CUDA Module"]:::device
            NUMA["NUMA Module"]:::device
            TFACC["TFACC Module"]:::device
            TOPS["TOPS Module"]:::device
        end
        CE -->|"uses"| CPU
        CE -->|"uses"| CUDA
        CE -->|"uses"| NUMA
        CE -->|"uses"| TFACC
        CE -->|"uses"| TOPS
    end

    subgraph "Layer 2: Model Handling & Conversion"
        MHC["Model Loader & Converter"]:::conversion
    end
    MHC -->|"prepares_models_for"| CE

    subgraph "Layer 3: API/Interface Layers"
        CLI["CLI Interface"]:::interface
        API["API Server/Web UI"]:::interface
        PY["Python Bindings"]:::interface
    end
    CLI -->|"invokes"| CE
    API -->|"invokes"| CE
    PY -->|"invokes"| CE

    subgraph "Layer 4: Example Applications"
        AND["Android Demo"]:::example
        WIN["Win32Demo (Desktop)"]:::example
        QUI["Qui Demo (Desktop)"]:::example
        APS["API Server Example"]:::example
        BEN["Benchmark Example"]:::example
    end
    CLI -->|"demonstrates"| AND
    API -->|"demonstrates"| WIN
    API -->|"demonstrates"| QUI
    API -->|"demonstrates"| APS
    CLI -->|"demonstrates"| BEN

    subgraph "Deployment & Utility Tools"
        CMake["CMake & Build Scripts"]:::utility
        Docker["Docker & CI/CD"]:::utility
    end
    CMake -->|"builds"| CE
    Docker -->|"deploys"| CE

    click CE "https://github.com/ztxz16/fastllm/blob/master/src/fastllm.cpp"
    click CPU "https://github.com/ztxz16/fastllm/tree/master/src/devices/cpu"
    click CUDA "https://github.com/ztxz16/fastllm/tree/master/src/devices/cuda"
    click NUMA "https://github.com/ztxz16/fastllm/tree/master/src/devices/numa"
    click TFACC "https://github.com/ztxz16/fastllm/tree/master/src/devices/tfacc"
    click TOPS "https://github.com/ztxz16/fastllm/tree/master/src/devices/tops"
    click MHC "https://github.com/ztxz16/fastllm/blob/master/tools/fastllm_pytools/export.py"
    click CLI "https://github.com/ztxz16/fastllm/tree/master/example/FastllmStudio/cli"
    click API "https://github.com/ztxz16/fastllm/tree/master/example/openai_server"
    click PY "https://github.com/ztxz16/fastllm/tree/master/pyfastllm"
    click AND "https://github.com/ztxz16/fastllm/tree/master/example/Android"
    click WIN "https://github.com/ztxz16/fastllm/tree/master/example/Win32Demo"
    click QUI "https://github.com/ztxz16/fastllm/tree/master/example/Qui"
    click APS "https://github.com/ztxz16/fastllm/tree/master/example/apiserver"
    click BEN "https://github.com/ztxz16/fastllm/tree/master/example/benchmark"
    click CMake "https://github.com/ztxz16/fastllm/blob/master/CMakeLists.txt"
    click Docker "https://github.com/ztxz16/fastllm/tree/master/.github/workflows"

    classDef core fill:#ADD8E6,stroke:#000,stroke-width:2px
    classDef device fill:#90EE90,stroke:#000,stroke-width:2px
    classDef conversion fill:#FFD700,stroke:#000,stroke-width:2px
    classDef interface fill:#FFA500,stroke:#000,stroke-width:2px
    classDef example fill:#FFB6C1,stroke:#000,stroke-width:2px
    classDef utility fill:#D3D3D3,stroke:#000,stroke-width:2px
```
