# flagos-ai/flaggems architecture

> Architecture diagram of the GitHub repository flagos-ai/flaggems, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/flagos-ai/flaggems
- Repository: https://github.com/flagos-ai/flaggems

## Overview

FlagGems is a Triton-based, PyTorch-compatible operator library for accelerating training and inference across accelerator backends. Its principal workflow is a PyTorch user invoking familiar APIs, FlagGems registering implementations with ATen, and the selected operator running a kernel to produce tensor results. Operator families include attention, fused LLM operations, and a broad general-purpose collection; runtime modules provide backend detection and tuning configuration. The tree also shows a substantial offline FlagTune workflow and a C++ dispatcher. Some subsystems were not sampled, so their internal links are omitted rather than inferred.

Main entry points: [`src/flag_gems/__init__.py`](https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/__init__.py) (FlagGems API), [`src/flag_gems/runtime/backend`](https://github.com/flagos-ai/flaggems/tree/master/src/flag_gems/runtime/backend) (Backend selection), [`cpp/include/flag_gems/operators.h`](https://github.com/flagos-ai/flaggems/blob/master/cpp/include/flag_gems/operators.h) (C++ operators), [`src/flag_gems/ops`](https://github.com/flagos-ai/flaggems/tree/master/src/flag_gems/ops) (General operators)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_integration["PyTorch integration"]
  node_api["PyTorch APIs"]
  node_package["FlagGems API<br/>[__init__.py]"]
  node_registrar["ATen registration<br/>[__init__.py]"]
end

subgraph group_kernels["Operator kernels"]
  node_general_ops["General operators"]
  node_attention["Attention kernels<br/>[attention.py]"]
  node_flash["Flash attention<br/>[flash_kernel.py]"]
  node_fused["Fused LLM ops"]
  node_moe["MoE routing<br/>[moe_routing.py]"]
  node_rope["RoPE kernels"]
end

subgraph group_runtime["Runtime dispatch"]
  node_backend["Backend selection"]
  node_device["Device detection<br/>[device_finder.py]"]
  node_config["Runtime config<br/>[configs_loader.py]"]
  node_dispatch["Kernel dispatch<br/>[libentry.py]"]
end

subgraph group_tuning["FlagTune"]
  node_flagtune_runtime["Tuning runtime<br/>[flagtune.py]"]
  node_offline["Offline tuning"]
  node_collect["Benchmark collection<br/>[scheduler.py]"]
  node_train["Route training<br/>[routed.py]"]
end

subgraph group_cpp["C++ interface"]
  node_cpp_bindings["C++ bindings<br/>[cstub.cpp]"]
  node_cpp_ops["C++ operators<br/>[operators.h]"]
  node_cpp_rope["C++ RoPE"]
  node_cpp_embedding["C++ embedding<br/>[embedding.cpp]"]
end

node_user(("PyTorch user"))

node_user -->|"invokes"| node_api
node_api -->|"uses"| node_package
node_package -->|"registers"| node_registrar
node_registrar -->|"registers"| node_general_ops
node_package -->|"initializes"| node_backend
node_backend -->|"detects"| node_device
node_backend -->|"loads"| node_config
node_general_ops -.->|"dispatches via"| node_dispatch
node_attention -->|"calls"| node_flash
node_package -->|"exports"| node_fused
node_package -->|"exposes"| node_moe
node_moe -->|"uses top-k"| node_fused
node_package -->|"exposes"| node_flagtune_runtime
node_cpp_bindings -->|"binds"| node_cpp_ops
node_cpp_ops -->|"declares"| node_cpp_rope
node_cpp_ops -->|"declares"| node_cpp_embedding
node_package -->|"exports"| node_rope
node_flagtune_runtime -->|"uses config"| node_config

click node_package "https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/__init__.py"
click node_registrar "https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/__init__.py"
click node_general_ops "https://github.com/flagos-ai/flaggems/tree/master/src/flag_gems/ops"
click node_attention "https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/ops/attention.py"
click node_flash "https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/ops/flash_kernel.py"
click node_fused "https://github.com/flagos-ai/flaggems/tree/master/src/flag_gems/fused"
click node_moe "https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/pt2/moe_routing.py"
click node_rope "https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/fused/rotary_embedding.py"
click node_backend "https://github.com/flagos-ai/flaggems/tree/master/src/flag_gems/runtime/backend"
click node_device "https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/runtime/backend/device_finder.py"
click node_config "https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/runtime/configs_loader.py"
click node_dispatch "https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/utils/libentry.py"
click node_flagtune_runtime "https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/runtime/flagtune.py"
click node_offline "https://github.com/flagos-ai/flaggems/tree/master/src/flag_gems/flagtune/offline"
click node_collect "https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/flagtune/offline/collection/scheduler.py"
click node_train "https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/flagtune/offline/train/routed.py"
click node_cpp_bindings "https://github.com/flagos-ai/flaggems/blob/master/cpp/csrc/cstub.cpp"
click node_cpp_ops "https://github.com/flagos-ai/flaggems/blob/master/cpp/include/flag_gems/operators.h"
click node_cpp_rope "https://github.com/flagos-ai/flaggems/blob/master/cpp/lib/rotary_embedding.cpp"
click node_cpp_embedding "https://github.com/flagos-ai/flaggems/blob/master/cpp/lib/embedding.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_package,node_registrar,node_user toneBlue
class node_general_ops,node_attention,node_flash,node_fused,node_moe,node_rope toneAmber
class node_backend,node_device,node_config,node_dispatch toneMint
class node_flagtune_runtime,node_offline,node_collect,node_train toneRose
class node_cpp_bindings,node_cpp_ops,node_cpp_rope,node_cpp_embedding toneIndigo
```

## Components

### PyTorch integration

- **PyTorch APIs**
- [FlagGems API](https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/__init__.py): `src/flag_gems/__init__.py`
- [ATen registration](https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/__init__.py): `src/flag_gems/__init__.py`

### Operator kernels

- [General operators](https://github.com/flagos-ai/flaggems/tree/master/src/flag_gems/ops): `src/flag_gems/ops`
- [Attention kernels](https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/ops/attention.py): `src/flag_gems/ops/attention.py`
- [Flash attention](https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/ops/flash_kernel.py): `src/flag_gems/ops/flash_kernel.py`
- [Fused LLM ops](https://github.com/flagos-ai/flaggems/tree/master/src/flag_gems/fused): `src/flag_gems/fused`
- [MoE routing](https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/pt2/moe_routing.py): `src/flag_gems/pt2/moe_routing.py`
- [RoPE kernels](https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/fused/rotary_embedding.py): `src/flag_gems/fused/rotary_embedding.py`

### Runtime dispatch

- [Backend selection](https://github.com/flagos-ai/flaggems/tree/master/src/flag_gems/runtime/backend): `src/flag_gems/runtime/backend`
- [Device detection](https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/runtime/backend/device_finder.py): `src/flag_gems/runtime/backend/device_finder.py`
- [Runtime config](https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/runtime/configs_loader.py): `src/flag_gems/runtime/configs_loader.py`
- [Kernel dispatch](https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/utils/libentry.py): `src/flag_gems/utils/libentry.py`

### FlagTune

- [Tuning runtime](https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/runtime/flagtune.py): `src/flag_gems/runtime/flagtune.py`
- [Offline tuning](https://github.com/flagos-ai/flaggems/tree/master/src/flag_gems/flagtune/offline): `src/flag_gems/flagtune/offline`
- [Benchmark collection](https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/flagtune/offline/collection/scheduler.py): `src/flag_gems/flagtune/offline/collection/scheduler.py`
- [Route training](https://github.com/flagos-ai/flaggems/blob/master/src/flag_gems/flagtune/offline/train/routed.py): `src/flag_gems/flagtune/offline/train/routed.py`

### C++ interface

- [C++ bindings](https://github.com/flagos-ai/flaggems/blob/master/cpp/csrc/cstub.cpp): `cpp/csrc/cstub.cpp`
- [C++ operators](https://github.com/flagos-ai/flaggems/blob/master/cpp/include/flag_gems/operators.h): `cpp/include/flag_gems/operators.h`
- [C++ RoPE](https://github.com/flagos-ai/flaggems/blob/master/cpp/lib/rotary_embedding.cpp): `cpp/lib/rotary_embedding.cpp`
- [C++ embedding](https://github.com/flagos-ai/flaggems/blob/master/cpp/lib/embedding.cpp): `cpp/lib/embedding.cpp`

### Other components

- **PyTorch user**

## Connections

- PyTorch user → PyTorch APIs: invokes (evidence: `README.md`)
- PyTorch APIs → FlagGems API: uses (evidence: `README.md`)
- FlagGems API → ATen registration: registers (evidence: `src/flag_gems/__init__.py`)
- ATen registration → General operators: registers (evidence: `src/flag_gems/__init__.py`)
- FlagGems API → Backend selection: initializes (evidence: `src/flag_gems/__init__.py`)
- Backend selection → Device detection: detects (evidence: `src/flag_gems/runtime/__init__.py`)
- Backend selection → Runtime config: loads (evidence: `src/flag_gems/runtime/__init__.py`)
- General operators → Kernel dispatch: dispatches via (evidence: `src/flag_gems/ops/attention.py`)
- Attention kernels → Flash attention: calls (evidence: `src/flag_gems/ops/attention.py`)
- FlagGems API → Fused LLM ops: exports (evidence: `src/flag_gems/__init__.py`)
- FlagGems API → MoE routing: exposes (evidence: `src/flag_gems/pt2/moe_routing.py`)
- MoE routing → Fused LLM ops: uses top-k (evidence: `src/flag_gems/pt2/moe_routing.py`)
- FlagGems API → Tuning runtime: exposes (evidence: `src/flag_gems/__init__.py`)
- C++ bindings → C++ operators: binds (evidence: `cpp/csrc/cstub.cpp`)
- C++ operators → C++ RoPE: declares (evidence: `cpp/include/flag_gems/operators.h`)
- C++ operators → C++ embedding: declares (evidence: `cpp/include/flag_gems/operators.h`)
- FlagGems API → RoPE kernels: exports (evidence: `src/flag_gems/__init__.py`)
- Tuning runtime → Runtime config: uses config (evidence: `src/flag_gems/runtime/__init__.py`)
