# foundation-model-stack/fms-model-optimizer architecture

> Architecture diagram of the GitHub repository foundation-model-stack/fms-model-optimizer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/foundation-model-stack/fms-model-optimizer
- Repository: https://github.com/foundation-model-stack/fms-model-optimizer

## Overview

##### Project type and purpose
- This is a Python‑based **model optimizer/quantization framework** built on top of PyTorch.
- Its purpose is to apply various quantization techniques (PTQ, QAT, GPTQ, FP8, etc.) to deep neural‑network models, optionally using custom kernels or third‑party addons.

##### File structure
- Top‑level directories:
- **fms_mo/** – the core package
- **build/** – Dockerfile and launch scripts for packaging/deployment
- **examples/** & **tutorials/** – usage demos and notebooks
- **tests/** – unit tests & reference artifacts
- Key subpackages under **fms_mo/**:
- **quant/** – quantization orchestration (ptq.py, quantizers.py)
- **dq.py** / **calib.py** – data‑quantization and calibration utilities
- **fx/** – PyTorch‑FX utilities for graph transforms
- **modules/** – implementations of quantized layers (conv, bmm, linear, lstm)
- **aiu_addons/** – plugin adapters for GPTQ, i8i8, etc.
- **custom_ext_kernels/** – custom CUDA/triton/CUTLASS kernels
- **recipes/** – JSON files driving configurable pipelines
- **utils/** – config, logging, dataset, TorchScript, error‑logging helpers
- **run_quant.py** – CLI / entrypoint that ties everything together
- Build and CI:
- **.github/** – workflows for lint, test, coverage, packaging

##### README insights
- Highlights Python API & small code changes to trigger large‑scale module/function replacement.
- Mentions optional external dependencies: auto_gptq, CUTLASS, Triton, matplotlib, Graphviz.

A. **User/Application**
– Scripts or notebooks call `run_quant.py` or import the Python API.
B. **Configuration Layer**
– JSON recipes (`fms_mo/recipes/*.json`)
– Command‑line args (`training_args.py`)
C. **Quantization Orchestrator**
– `run_quant.py` → loads model, parses config
– Invokes **Prep** → **FX transforms** → **Calibration** → **Quantizers** → **Module Replacement**
D. **Core Library Modules**
1. **fx/**: Graph extraction & transformation
2. **calib.py** / **dq.py**: Data collection & scale computation
3. **quant/**: Quantizer classes apply quant/dequant ops
4. **modules/**: Quantized layer implementations
E. **Extension Points (Plugin Pattern)**
– **aiu_addons/**: Adapter interfaces for GPTQ & i8i8
– **custom_ext_kernels/**: High‑performance CUDA/CUTLASS/Triton kernels
F. **Utilities**
– Logging, error handling, dataset loaders, TorchScript helpers
G. **External Dependencies & Hardware**
– PyTorch (Dynamo, FX)
– CUDA / GPU hardware (V100/A100/H100)
– Optional: auto_gptq, CUTLASS repo, Triton
– Visualization: matplotlib, graphviz

##### Relationships & data flow
1. **User** → triggers → **run_quant.py**
2. **run_quant** reads **config/recipes**, instantiates **training_args**
3. Calls **fms_mo.prep** to prepare model & data
4. Passes model through **fx** utilities → returns an FX graph module
5. **calib** collects activations using sample data → scale factors
6. **quant** classes inject quant/dequant ops & replace original modules
7. **aiu_addons** or **custom_ext_kernels** are plugged in for specific ops
8. The result is a quantized model; optionally **TorchScript** or **export**
9. **User** consumes the optimized model

##### Architectural patterns & design principles
- **Plugin/Adapter pattern** for addons & custom kernels
- **Pipeline pattern**: clear stages (prep → fx → calib → quant → replace)
- **Recipe‑driven configuration** decouples algorithm choices from code
- **Layered modularity**: separation between graph logic, quant logic, kernels

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User/Application Entry
    User["User / Notebook / Script"]:::external
    Entry["run_quant.py (CLI & API)"]:::core
    click Entry "https://github.com/foundation-model-stack/fms-model-optimizer/blob/main/fms_mo/run_quant.py"
    User -->|"triggers"| Entry

    %% Configuration Layer
    subgraph "Configuration Layer"
        Recipes["JSON Recipes"]:::config
        click Recipes "https://github.com/foundation-model-stack/fms-model-optimizer/blob/main/fms_mo/recipes/*.json"
        Args["Training Args"]:::config
        click Args "https://github.com/foundation-model-stack/fms-model-optimizer/blob/main/fms_mo/training_args.py"
    end
    Entry -->|"reads recipes & args"| Recipes
    Entry -->|"reads recipes & args"| Args

    %% Quantization Orchestrator Pipeline
    subgraph "Quantization Orchestrator" 
        Prep["Prep"]:::core
        click Prep "https://github.com/foundation-model-stack/fms-model-optimizer/blob/main/fms_mo/prep.py"
        FX["FX Graph Utils"]:::core
        click FX "https://github.com/foundation-model-stack/fms-model-optimizer/tree/main/fms_mo/fx/"
        Calib["Calibration & DQ"]:::core
        click Calib "https://github.com/foundation-model-stack/fms-model-optimizer/blob/main/fms_mo/calib.py"
        DQ["Data‑Quant Utilities"]:::core
        click DQ "https://github.com/foundation-model-stack/fms-model-optimizer/blob/main/fms_mo/dq.py"
        Quant["Quantizer Classes"]:::core
        click Quant "https://github.com/foundation-model-stack/fms-model-optimizer/blob/main/fms_mo/quant/ptq.py"
        click Quant "https://github.com/foundation-model-stack/fms-model-optimizer/blob/main/fms_mo/quant/quantizers.py"
        Replace["Module Replacement"]:::core
        click Replace "https://github.com/foundation-model-stack/fms-model-optimizer/tree/main/fms_mo/modules/"
        Torch["TorchScript / Export"]:::core
    end
    Entry --> Prep
    Prep -->|"model & data"| FX
    FX -->|"FX graph"| Calib
    Calib -->|"scale factors"| DQ
    DQ --> Quant
    Quant -->|"quantized ops"| Replace
    Replace --> Torch
    Torch --> QModel

    %% Extension Points
    subgraph "Extension Points"
        AIU["AIU Addons (GPTQ / i8i8)"]:::extension
        click AIU "https://github.com/foundation-model-stack/fms-model-optimizer/tree/main/fms_mo/aiu_addons/"
        Kernels["Custom Ext Kernels"]:::extension
        click Kernels "https://github.com/foundation-model-stack/fms-model-optimizer/tree/main/fms_mo/custom_ext_kernels/"
    end
    Quant -->|"plugins"| AIU
    Quant -->|"plugins"| Kernels
    Replace --> AIU
    Replace --> Kernels

    %% Utilities
    Utils["Utilities"]:::utils
    click Utils "https://github.com/foundation-model-stack/fms-model-optimizer/tree/main/fms_mo/utils/"
    Prep --> Utils
    FX --> Utils
    Calib --> Utils
    Quant --> Utils
    Replace --> Utils

    %% Output Model and Consumption
    QModel["Quantized Model"]:::core
    Torch -->|"outputs model"| QModel
    QModel -->|"consumes"| User

    %% External Dependencies & Infrastructure
    subgraph "External Dependencies & Hardware"
        PyTorch["PyTorch (Dynamo, FX)"]:::external
        CUDA["CUDA / GPU (V100/A100/H100)"]:::external
        OptLibs["Optional Libs\n(auto_gptq, CUTLASS, Triton)"]:::external
        Viz["Visualization\n(matplotlib, graphviz)"]:::external
    end
    FX --> PyTorch
    Calib --> PyTorch
    Quant --> PyTorch
    Replace --> CUDA
    AIU --> OptLibs
    Kernels --> CUDA
    Torch --> PyTorch
    Torch --> CUDA
    Torch --> Viz

    %% Examples, Build & Docs
    Examples["Examples & Tutorials"]:::external
    click Examples "https://github.com/foundation-model-stack/fms-model-optimizer/tree/main/examples/"
    click Examples "https://github.com/foundation-model-stack/fms-model-optimizer/tree/main/tutorials/"
    Build["Build & Deployment Scripts"]:::external
    click Build "https://github.com/foundation-model-stack/fms-model-optimizer/tree/main/build/"
    Docs["Documentation & Design Diagrams"]:::external
    click Docs "https://github.com/foundation-model-stack/fms-model-optimizer/tree/main/docs/"
    Examples --> User
    Build --> Entry
    Docs --> User

    %% Styles
    classDef core fill:#AED6F1,stroke:#21618C,color:#1B4F72
    classDef extension fill:#ABEBC6,stroke:#239B56,color:#196F3D
    classDef external fill:#F9E79F,stroke:#B7950B,color:#7D6608
    classDef config fill:#D5D8DC,stroke:#7B7D7D,color:#424949
    classDef utils fill:#FADBD8,stroke:#C0392B,color:#78281F
```
