# quic/efficient-transformers architecture

> Architecture diagram of the GitHub repository quic/efficient-transformers, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/quic/efficient-transformers
- Repository: https://github.com/quic/efficient-transformers

## Overview

##### Project Type & Purpose
- Type: Open-source Python library/toolchain.
- Purpose: Reimplement, transform, compile, export, fine-tune, and run LLMs efficiently on Qualcomm Cloud AI 100 hardware using ONNX, PyTorch, custom operators, graph transforms, and integration with HuggingFace Transformers.

##### Key Architectural Components
A. Python API Layer (QEfficient package)
– Subpackages:
- base (common utilities, modeling primitives)
- transformers (HuggingFace model wrappers, quantizers, samplers, SPD)
- peft (adapter/LoRA support)
– Utilities: QEfficient/utils (logging, caching, device management, hash, checkpoint)
B. Graph Transformation & Compilation
– compile: compile_helper.py, qnn_compiler.py, qnn_config.json
– customop: specialized CUDA/ONNX kernels (ctx_scatter_gather, matmulnbits, rms_norm)
C. Exporter
– exporter: export_hf_to_cloud_ai_100.py, export_utils.py → ONNX graph output
D. Cloud Integration
– cloud: compile.py, export.py, finetune.py, execute.py, infer.py → wraps Qualcomm Cloud AI SDK/API
E. Finetuning & Training
– finetune: configs, data (sampler, dataset wrappers), eval, utils → trains/adapts LLMs
F. Inference & Generation
– generation: cloud_infer.py, text_generation_inference.py → model invocation pipelines
G. External Dependencies
– HuggingFace Transformers
– PyTorch
– ONNX & ONNX Runtime
– Qualcomm Cloud AI 100 SDK
– CI/CD (GitHub Actions, pre-commit)
H. Examples, Tests, Docs, Scripts
– examples/ (python, C++), notebooks/
– tests/ (unit, integration)
– docs/ (Sphinx)
– scripts/ (finetune runners, perplexity, kv replication)

##### Interactions & Data Flows
- User code or example script invokes QEfficient API → Model wrappers in transformers/peft →
→ Optionally finetune via finetune/ → Trained model objects → export via exporter → ONNX graph →
→ compile via compile/ → compiled artifact → push to Cloud AI 100 via cloud/ → inference via cloud/
→ results back to user.
- CustomOp modules get registered at graph-compile or runtime to replace Transformers ops.
- Utilities provide cross-cutting concerns (logging, device selection, caching).
- CI pipelines ensure code quality, formatting, and run tests against base, cloud, compile, peft, transformers modules.

##### Architectural Patterns & Design Principles
- Layered modular architecture: clear separation of API, transformation/compilation, exporter, cloud integration, training, inference.
- Plugin/extension points: customop modules, quantizer auto-registration, dataset/config plugins for finetuning.
- Pipeline style: sequential stages (model load → transform → export → compile → deploy → infer).
- Dependency inversion: high-level API relies on abstractions (common interfaces), concrete implementations plugged in (e.g., QNN compiler).
- Reuse of external frameworks: huggingface transformers for modeling, ONNX/ORT for runtime.

##### Essential Details to Capture
- Modular packages and submodules
- Pipeline order: load → transform → export → compile → deploy → infer
- Extension points (customop, quantizers, dataset plugins)
- Utility services (logging, caching, device management)
- External frameworks and cloud SDK

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Entry
    User["User Script/Example"]:::external

    %% API Layer
    subgraph "Python API Layer" 
        direction TB
        Base["Base Module"]:::core
        Transformers["Transformers Module"]:::core
        PEFT["PEFT Module"]:::core
        Utils["Utilities (logging, caching, device mgmt)"]:::core
    end

    %% Finetuning & Training
    subgraph "Finetuning & Training"
        direction TB
        FT_Configs["Configs"]:::core
        FT_Data["Data Sampling"]:::core
        FT_Dataset["Dataset Wrappers"]:::core
        FT_EvalUtils["Eval & Utils"]:::core
    end

    %% Graph Transform & Compilation
    subgraph "Graph Transformation & Compilation"
        direction TB
        CompileHelper["compile_helper.py"]:::core
        QNNCompiler["qnn_compiler.py"]:::core
        QNNConfig["qnn_config.json"]:::core
        CustomOp["Custom Operators"]:::core
    end

    %% Exporter
    subgraph "Exporter"
        direction TB
        ExportScript["export_hf_to_cloud_ai_100.py"]:::core
        ExportUtils["export_utils.py"]:::core
    end

    %% Artifacts
    subgraph "Data Artifacts"
        direction TB
        ModelObj["Trained Model Object"]:::data
        ONNXGraph["ONNX Graph"]:::data
        CompiledArtifact["Compiled Artifact"]:::data
    end

    %% Cloud Integration
    subgraph "Cloud Integration"
        direction TB
        CloudCompile["compile.py"]:::core
        CloudExport["export.py"]:::core
        CloudFinetune["finetune.py"]:::core
        CloudExecute["execute.py"]:::core
        CloudInfer["infer.py"]:::core
    end

    %% Inference & Generation
    subgraph "Inference & Generation"
        direction TB
        CloudInferPipe["cloud_infer.py"]:::core
        TextGenInfer["text_generation_inference.py"]:::core
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        HF["HuggingFace Transformers"]:::external
        Torch["PyTorch"]:::external
        ONNXRuntime["ONNX & ONNX Runtime"]:::external
        CloudSDK["Qualcomm Cloud AI 100 SDK"]:::external
    end

    %% Support
    subgraph "Examples, Tests, Docs, Scripts"
        direction TB
        Examples["examples/"]:::core
        Tests["tests/"]:::core
        Docs["docs/"]:::core
        Scripts["scripts/"]:::core
    end

    %% Data Flow
    User -->|imports| Base
    Base --> Transformers
    Base --> PEFT
    Transformers --> Utils
    PEFT --> Utils

    Transformers -->|load model| ModelObj
    ModelObj --> FT_Configs
    ModelObj --> FT_Data
    ModelObj --> FT_Dataset
    FT_EvalUtils --> ModelObj

    ModelObj --> ExportScript
    ExportScript --> ONNXGraph
    ExportUtils --> ONNXGraph

    ONNXGraph --> CompileHelper
    CompileHelper --> QNNCompiler
    QNNConfig --> QNNCompiler
    CustomOp --> QNNCompiler
    QNNCompiler --> CompiledArtifact

    CompiledArtifact --> CloudCompile
    CompiledArtifact --> CloudExport
    CloudCompile --> CloudSDK
    CloudExport --> CloudSDK

    CloudFinetune --> CloudSDK
    CloudExecute --> CloudSDK
    CloudInfer --> CloudSDK

    CloudCompile --> CloudInferPipe
    CloudExport --> TextGenInfer
    CloudInferPipe --> Results["Generation Results"]:::data
    TextGenInfer --> Results

    %% External integrations
    Transformers --> HF
    PEFT --> HF
    ONNXGraph --> ONNXRuntime
    CompiledArtifact --> CloudSDK

    %% Support links
    User --> Examples
    User --> Tests
    User --> Docs
    User --> Scripts

    %% Click Events
    click Base "https://github.com/quic/efficient-transformers/tree/main/QEfficient/base/"
    click Transformers "https://github.com/quic/efficient-transformers/tree/main/QEfficient/transformers/"
    click PEFT "https://github.com/quic/efficient-transformers/tree/main/QEfficient/peft/"
    click Utils "https://github.com/quic/efficient-transformers/tree/main/QEfficient/utils/"
    click CompileHelper "https://github.com/quic/efficient-transformers/blob/main/QEfficient/compile/compile_helper.py"
    click QNNCompiler "https://github.com/quic/efficient-transformers/blob/main/QEfficient/compile/qnn_compiler.py"
    click QNNConfig "https://github.com/quic/efficient-transformers/blob/main/QEfficient/compile/qnn_config.json"
    click CustomOp "https://github.com/quic/efficient-transformers/tree/main/QEfficient/customop/"
    click ExportScript "https://github.com/quic/efficient-transformers/blob/main/QEfficient/exporter/export_hf_to_cloud_ai_100.py"
    click ExportUtils "https://github.com/quic/efficient-transformers/blob/main/QEfficient/exporter/export_utils.py"
    click CloudCompile "https://github.com/quic/efficient-transformers/blob/main/QEfficient/cloud/compile.py"
    click CloudExport "https://github.com/quic/efficient-transformers/blob/main/QEfficient/cloud/export.py"
    click CloudFinetune "https://github.com/quic/efficient-transformers/blob/main/QEfficient/cloud/finetune.py"
    click CloudExecute "https://github.com/quic/efficient-transformers/blob/main/QEfficient/cloud/execute.py"
    click CloudInfer "https://github.com/quic/efficient-transformers/blob/main/QEfficient/cloud/infer.py"
    click FT_Configs "https://github.com/quic/efficient-transformers/tree/main/QEfficient/finetune/configs/"
    click FT_Data "https://github.com/quic/efficient-transformers/tree/main/QEfficient/finetune/data/"
    click FT_Dataset "https://github.com/quic/efficient-transformers/tree/main/QEfficient/finetune/dataset/"
    click FT_EvalUtils "https://github.com/quic/efficient-transformers/blob/main/QEfficient/finetune/eval.py"
    click FT_EvalUtils "https://github.com/quic/efficient-transformers/tree/main/QEfficient/finetune/utils/"
    click CloudInferPipe "https://github.com/quic/efficient-transformers/blob/main/QEfficient/generation/cloud_infer.py"
    click TextGenInfer "https://github.com/quic/efficient-transformers/blob/main/QEfficient/generation/text_generation_inference.py"
    click Examples "https://github.com/quic/efficient-transformers/tree/main/examples/"
    click Tests "https://github.com/quic/efficient-transformers/tree/main/tests/"
    click Docs "https://github.com/quic/efficient-transformers/tree/main/docs/"
    click Scripts "https://github.com/quic/efficient-transformers/tree/main/scripts/"

    %% Styles
    classDef core fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef data fill:#90EE90,stroke:#000,stroke-width:1px
    classDef external fill:#FFA500,stroke:#000,stroke-width:1px
```
