How orcnnn/monitoring works
Project Type and Purpose
• SWIFT is an open-source Python framework and CLI tool (ms-swift) for the entire large language/multimodal model pipeline: data loading, pre-training, fine-tuning, human alignment (RLHF), inference, evaluation, quantization, deployment, and a Gradio-based Web-UI.
• It integrates with external compute backends (PyTorch, vLLM, LMDeploy), evaluation backends (EvalScope), model/dataset hubs (ModelScope), and hardware platforms (GPU, NPU, CPU).
High-Level Components
A. Command-Line Interface (“swift” CLI)
• Entry point: swift/cli/main.py
• Subcommands: sft, pt, rlhf, infer, eval, export, deploy, web-ui
• Parses arguments → dispatches to corresponding module’s “app.py” or handler
B. Core Python Library (“swift” Package)
• Dataset layer (swift/llm/dataset): data loaders, preprocessors, registry
• Model layer (swift/llm/model): model factory, patchers, support for 450+ models
• Template layer (swift/llm/template): prompt templates, registry
• Training layer (swift/llm/train + swift/trainers + swift/tuners):
• Pre-training (pt), fine-tuning (sft), RLHF trainers, plugin/extensibility
• Distributed strategies: DDP, ZeRO, FSDP, device_map
• Tuners: LoRA, QLoRA, LlamaPro, LongLoRA, and dozens of others
• Inference layer (swift/llm/infer): multiple engine implementations (PtEngine, vLLM, LMDeploy, VLLM, LMDeploy) behind a common “InferEngine” protocol
• Export/Quantization layer (swift/llm/export): GPTQ, AWQ, BNB, merging lora, Ollama export
• Evaluation layer (swift/llm/eval): EvalScope backend integration
• Deployment layer (swift/cli/deploy + swift/llm/infer/deploy): server startup scripts for client-server pattern
• Plugin layer (swift/plugin): hooks for custom loss, optimizer, metrics, callbacks
• Utilities (swift/utils): IO, logging, environment detection, tensor/torch wrappers
C. Web-UI Layer (swift/ui)
• Gradio-based interactive UIs for train, infer, eval, export
• Modules: llm_train, llm_infer, llm_eval, llm_export
D. Examples & Scripts
• examples/: concrete shell & Python demos showing CLI usage, notebook samples
• scripts/: benchmarking, reporting, dataset/model info
External Dependencies & Infrastructure
• Compute Backends: PyTorch, vLLM, LMDeploy
• Model/Dataset Hub: ModelScope (via swift/hub)
• Hardware: CPU, GPU (NVIDIA A100/V100/T4), Ascend NPU
• Evaluation: EvalScope
• Packaging/CI: Docker, GitHub Actions, pre-commit, setup.py
Relationships and Data Flows
• User invokes “swift <command>” → CLI parser → calls core library APIs
• Core library orchestrates:
• Dataset layer → yields TokenizedDataset
• Template layer → wraps data into prompt format
• Training layer → uses Trainer Factory to instantiate correct Trainer → trains model on hardware
• Inference layer → creates InferEngine → sends InferRequest → returns InferResponse
• Export layer → reads model artifacts → quantizes/merges → writes output
• Evaluation layer → sends test data → collects metrics → generates report
• Deployment layer → launches server (HTTP/gRPC) → clients connect
• Web-UI wraps the same core APIs behind a Gradio interface
Architectural Patterns & Principles
• Modular, Layered Architecture: clear separation of concerns by function (dataset, model, train, infer, export, eval, deploy)
• Plugin/Extension Points: registry patterns for models, datasets, templates, tuners, optimizers, callbacks
• Factory/Strategy Patterns: TrainerFactory, InferEngine interface, multiple backend implementations
• CLI-Library Duality: CLI commands are thin wrappers over Python API
• Declarative Configurations: JSON/YAML for ds_config, trainer, device_map, CI