# sjtu-ipads/powerinfer architecture

> Architecture diagram of the GitHub repository sjtu-ipads/powerinfer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sjtu-ipads/powerinfer
- Repository: https://github.com/sjtu-ipads/powerinfer

## Overview

##### Project Components Identification
- Core Inference Engine: The heart of the project is the fast inference engine written mainly in C/C++ (see files like llama.cpp, ggml.c/h, common/*, and tests/*). This module implements the LLM serving logic, including the hybrid CPU/GPU scheduling (hot/cold neuron offloading).
- Model Conversion & Weight Processing: There are several Python and C++ scripts (convert.py, convert-dense.py, convert-hf-to-powerinfer-gguf.py) that help in converting original model weights (and predictor weights) into the PowerInfer GGUF format. This conversion layer is a key preprocessing component.
- GPU Offloading Module: Notice the CUDA, ROCm and related files (e.g., in .devops, ggml-cuda.cu, ggml-metal/*) indicating support for different GPU backends and specialized sparse operators.
- Command Line and Serving Examples: The examples/ directory contains several demonstration applications such as server, batched generation, perplexity evaluation, and chat demos. These act as frontends that use the core library.
- Supporting Utilities: This includes testing (tests/), documentation (docs/), Docker configurations (in .devops and .github/workflows), and helper tools for quantization and performance measurement.

A. Main Components and Their Relationships
– Core Inference Engine: Central block encapsulating the LLM inference algorithm and hybrid CPU/GPU scheduling. Subdivide this into:
- CPU Processing Unit: For “cold” neurons and non-critical processing.
– Model Conversion Module: Place a separate block on the left (or the entry point) representing conversion/quantization tools that process raw model weights into the “PowerInfer GGUF” format.
– Data & Model Repository: Illustrate a storage component where model weights, predictor weights, and activation statistics reside. This highlights the importance of file structure expected by the inference engine.

B. Communication and Data Flow:
- The initial processing of raw model data through the Model Conversion Module.
- The parsed/converted model then gets loaded by the Core Inference Engine.
- Frontend examples (CLI, server, etc.) invoke the inference engine to generate tokens.

C. Architectural Patterns and Technologies:
– Emphasize the separation of concerns:
- The conversion/quantization process is decoupled from the inference runtime.
- The hybrid compute architecture distinguishes between CPU-bound and GPU-bound work.
– Highlight the use of CMake for builds, Docker for containerized deployment, and Python scripts for conversion utilities.
– Note influences from llama.cpp and ggml for lower-level efficient matrix computations.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Preprocessing & Data Layer
    subgraph "Preprocessing & Data Layer"
        subgraph "Model Conversion Tools"
            CP1["convert.py"]:::conversion
            CP2["convert-dense.py"]:::conversion
            CP3["convert-hf-to-powerinfer-gguf.py"]:::conversion
            CP4["convert-llama2c-to-ggml.cpp"]:::conversion
        end
        DM["Data & Model Repository"]:::database
    end

    %% Core Engine Layer
    subgraph "Core Engine"
        subgraph "Inference Core Code"
            Llama["llama.cpp"]:::core
            GGML["ggml.c & ggml.h"]:::core
            Common["common/ Files"]:::core
            Tests["tests/ Files"]:::core
        end
        subgraph "Hybrid Scheduler"
            CPU["CPU Processing Unit"]:::cpu
            subgraph "GPU Offloading Module"
                CudaC["ggml-cuda.cu"]:::gpu
                CudaH["ggml-cuda.h"]:::gpu
                Metal["ggml-metal Files"]:::gpu
                Docker["GPU Docker Configs"]:::gpu
            end
        end
    end

    %% User Interfaces
    subgraph "User Interfaces"
        UA["User Applications/Examples"]:::ui
    end

    %% Deployment & Supporting Utilities
    subgraph "Deployment & Support"
        BD["Build & Deployment Pipeline"]:::deployment
        SU["Supporting Utilities"]:::utilities
    end

    %% External Dependencies
    EXT["External Dependencies & Libraries"]:::external

    %% Data Flow Connections
    CP1 --> DM
    CP2 --> DM
    CP3 --> DM
    CP4 --> DM

    DM --> Llama
    DM --> GGML
    DM --> Common
    DM --> Tests

    Llama -->|"loads"| CPU
    GGML --> CPU
    Common --> CPU
    Tests --> CPU

    CPU --> UA
    Docker --> UA

    EXT --> Llama
    EXT --> GGML

    %% Connections between Core and Offloading
    CPU --- CudaC
    CPU --- CudaH
    CPU --- Metal
    CPU --- Docker

    %% Deployment and Support flows
    BD --> EXT
    BD --> SU
    SU --> UA

    %% Click Events for Model Conversion Tools
    click CP1 "https://github.com/sjtu-ipads/powerinfer/blob/main/convert.py"
    click CP2 "https://github.com/sjtu-ipads/powerinfer/blob/main/convert-dense.py"
    click CP3 "https://github.com/sjtu-ipads/powerinfer/blob/main/convert-hf-to-powerinfer-gguf.py"
    click CP4 "https://github.com/sjtu-ipads/powerinfer/blob/main/examples/convert-llama2c-to-ggml/convert-llama2c-to-ggml.cpp"

    %% Click Events for Core Inference Code
    click Llama "https://github.com/sjtu-ipads/powerinfer/blob/main/llama.cpp"
    click GGML "https://github.com/sjtu-ipads/powerinfer/blob/main/ggml.c"
    click Common "https://github.com/sjtu-ipads/powerinfer/tree/main/common/"
    click Tests "https://github.com/sjtu-ipads/powerinfer/tree/main/tests/"

    %% Click Events for GPU Offloading Module
    click CudaC "https://github.com/sjtu-ipads/powerinfer/blob/main/ggml-cuda.cu"
    click CudaH "https://github.com/sjtu-ipads/powerinfer/blob/main/ggml-cuda.h"
    click Metal "https://github.com/sjtu-ipads/powerinfer/blob/main/ggml-metal.h"
    click Docker "https://github.com/sjtu-ipads/powerinfer/blob/main/.devops/full-cuda.Dockerfile"

    %% Click Event for User Applications
    click UA "https://github.com/sjtu-ipads/powerinfer/tree/main/examples/"

    %% Click Events for Deployment & Supporting Utilities
    click BD "https://github.com/sjtu-ipads/powerinfer/tree/main/.devops/"
    click SU "https://github.com/sjtu-ipads/powerinfer/tree/main/docs/"

    %% Styling
    classDef conversion fill:#AED6F1,stroke:#1f618d,stroke-width:2px;
    classDef database fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;
    classDef core fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef cpu fill:#F5B7B1,stroke:#922B21,stroke-width:2px;
    classDef gpu fill:#A9DFBF,stroke:#1E8449,stroke-width:2px;
    classDef ui fill:#D4E6F1,stroke:#2874A6,stroke-width:2px;
    classDef deployment fill:#FAD7A0,stroke:#D68910,stroke-width:2px;
    classDef utilities fill:#FAD7A0,stroke:#D68910,stroke-width:2px;
    classDef external fill:#C39BD3,stroke:#6C3483,stroke-width:2px;
    
    %% Minor spacing tweaks
    style CP1 margin:10px
    style DM margin:10px
    style Llama margin:10px
    style UA margin:10px
    style BD margin:10px
    style SU margin:10px
```
