How foundation-model-stack/fms-model-optimizer works
Project type and purpose
• This is a Python‑based model optimizer/quantization framework built on top of PyTorch.
• Its purpose is to apply various quantization techniques (PTQ, QAT, GPTQ, FP8, etc.) to deep neural‑network models, optionally using custom kernels or third‑party addons.
File structure
• Top‑level directories:
• fms_mo/ – the core package
• build/ – Dockerfile and launch scripts for packaging/deployment
• examples/ & tutorials/ – usage demos and notebooks
• tests/ – unit tests & reference artifacts
• Key subpackages under fms_mo/:
• quant/ – quantization orchestration (ptq.py, quantizers.py)
• dq.py / calib.py – data‑quantization and calibration utilities
• fx/ – PyTorch‑FX utilities for graph transforms
• modules/ – implementations of quantized layers (conv, bmm, linear, lstm)
• aiu_addons/ – plugin adapters for GPTQ, i8i8, etc.
• custom_ext_kernels/ – custom CUDA/triton/CUTLASS kernels
• recipes/ – JSON files driving configurable pipelines
• utils/ – config, logging, dataset, TorchScript, error‑logging helpers
• run_quant.py – CLI / entrypoint that ties everything together
• Build and CI:
• .github/ – workflows for lint, test, coverage, packaging
README insights
• Highlights Python API & small code changes to trigger large‑scale module/function replacement.
• Mentions optional external dependencies: auto_gptq, CUTLASS, Triton, matplotlib, Graphviz.
A. User/Application
– Scripts or notebooks call run_quant.py or import the Python API.
B. Configuration Layer
– JSON recipes (fms_mo/recipes/*.json)
– Command‑line args (training_args.py)
C. Quantization Orchestrator
– run_quant.py → loads model, parses config
– Invokes Prep → FX transforms → Calibration → Quantizers → Module Replacement
D. Core Library Modules
1. fx/: Graph extraction & transformation
2. calib.py / dq.py: Data collection & scale computation
3. quant/: Quantizer classes apply quant/dequant ops
4. modules/: Quantized layer implementations
E. Extension Points (Plugin Pattern)
– aiu_addons/: Adapter interfaces for GPTQ & i8i8
– custom_ext_kernels/: High‑performance CUDA/CUTLASS/Triton kernels
F. Utilities
– Logging, error handling, dataset loaders, TorchScript helpers
G. External Dependencies & Hardware
– PyTorch (Dynamo, FX)
– CUDA / GPU hardware (V100/A100/H100)
– Optional: auto_gptq, CUTLASS repo, Triton
– Visualization: matplotlib, graphviz
Relationships & data flow
1. User → triggers → run_quant.py
2. run_quant reads config/recipes, instantiates training_args
3. Calls fms_mo.prep to prepare model & data
4. Passes model through fx utilities → returns an FX graph module
5. calib collects activations using sample data → scale factors
6. quant classes inject quant/dequant ops & replace original modules
7. aiu_addons or custom_ext_kernels are plugged in for specific ops
8. The result is a quantized model; optionally TorchScript or export
9. User consumes the optimized model
Architectural patterns & design principles
• Plugin/Adapter pattern for addons & custom kernels
• Pipeline pattern: clear stages (prep → fx → calib → quant → replace)
• Recipe‑driven configuration decouples algorithm choices from code
• Layered modularity: separation between graph logic, quant logic, kernels