How quic/efficient-transformers works
Project Type & Purpose
• Type: Open-source Python library/toolchain.
• Purpose: Reimplement, transform, compile, export, fine-tune, and run LLMs efficiently on Qualcomm Cloud AI 100 hardware using ONNX, PyTorch, custom operators, graph transforms, and integration with HuggingFace Transformers.
Key Architectural Components
A. Python API Layer (QEfficient package)
– Subpackages:
• base (common utilities, modeling primitives)
• transformers (HuggingFace model wrappers, quantizers, samplers, SPD)
• peft (adapter/LoRA support)
– Utilities: QEfficient/utils (logging, caching, device management, hash, checkpoint)
B. Graph Transformation & Compilation
– compile: compile_helper.py, qnn_compiler.py, qnn_config.json
– customop: specialized CUDA/ONNX kernels (ctx_scatter_gather, matmulnbits, rms_norm)
C. Exporter
– exporter: export_hf_to_cloud_ai_100.py, export_utils.py → ONNX graph output
D. Cloud Integration
– cloud: compile.py, export.py, finetune.py, execute.py, infer.py → wraps Qualcomm Cloud AI SDK/API
E. Finetuning & Training
– finetune: configs, data (sampler, dataset wrappers), eval, utils → trains/adapts LLMs
F. Inference & Generation
– generation: cloud_infer.py, text_generation_inference.py → model invocation pipelines
G. External Dependencies
– HuggingFace Transformers
– PyTorch
– ONNX & ONNX Runtime
– Qualcomm Cloud AI 100 SDK
– CI/CD (GitHub Actions, pre-commit)
H. Examples, Tests, Docs, Scripts
– examples/ (python, C++), notebooks/
– tests/ (unit, integration)
– docs/ (Sphinx)
– scripts/ (finetune runners, perplexity, kv replication)
Interactions & Data Flows
• User code or example script invokes QEfficient API → Model wrappers in transformers/peft →
→ Optionally finetune via finetune/ → Trained model objects → export via exporter → ONNX graph →
→ compile via compile/ → compiled artifact → push to Cloud AI 100 via cloud/ → inference via cloud/
→ results back to user.
• CustomOp modules get registered at graph-compile or runtime to replace Transformers ops.
• Utilities provide cross-cutting concerns (logging, device selection, caching).
• CI pipelines ensure code quality, formatting, and run tests against base, cloud, compile, peft, transformers modules.
Architectural Patterns & Design Principles
• Layered modular architecture: clear separation of API, transformation/compilation, exporter, cloud integration, training, inference.
• Plugin/extension points: customop modules, quantizer auto-registration, dataset/config plugins for finetuning.
• Pipeline style: sequential stages (model load → transform → export → compile → deploy → infer).
• Dependency inversion: high-level API relies on abstractions (common interfaces), concrete implementations plugged in (e.g., QNN compiler).
• Reuse of external frameworks: huggingface transformers for modeling, ONNX/ORT for runtime.
Essential Details to Capture
• Modular packages and submodules
• Pipeline order: load → transform → export → compile → deploy → infer
• Extension points (customop, quantizers, dataset plugins)
• Utility services (logging, caching, device management)
• External frameworks and cloud SDK