How jlin1201/llama-factory works
Main components and modules
• CLI and Command Line Interface – the entry point for training, evaluation, chat, exporting/merging models, etc. (located in the “src/llamafactory/cli.py” and top-level “train.py”).
• API Service – an OpenAI-style API server for inference (e.g., “src/api.py” and “src/llamafactory/api/…”). This module handles endpoints (chat, image testing, toolcall, etc.) and is responsible for interfacing with external applications.
• Web UI – a Gradio-powered GUI (under “src/llamafactory/webui”) that provides an interactive interface for fine-tuning and model inference.
• Training Subsystem – comprising multiple training workflows and fine-tuning strategies organized in “src/llamafactory/train”. Here you find dedicated submodules (for SFT, DPO, PPO, KTO, etc.) along with callback and workflow scripts.
• Data Processing and Loader – the “src/llamafactory/data” package that handles dataset loading, conversion, formatting, and collating. It supports both local data as well as integration with datasets on hubs.
• Model and Factory Layers – found in “src/llamafactory/model”, “src/llamafactory/llamafactory” (which includes modules for loading, patching, adapter layers, quantization, etc.). These components abstract over various LLMs and support adapters like LoRA, quantized training modules, and other model utilities.
• Evaluation Module – under “src/llamafactory/eval”, which defines evaluators, metrics, and provides interfaces to test aspects of fine-tuning.
• External Integrations and Extras – additional utilities (logging, hyperparameter parsing, misc constants, and extra algorithms in “src/llamafactory/extras” and “src/llamafactory/hparams”) that enable integrations with logging platforms (WandB, SwanLab), Docker deployments, and even specialized optimizers (GaLore, Adam-mini, etc.).
Relationships and data flows
• The CLI acts as an orchestrator and selects a YAML configuration that specifies a chosen training routine or inference setup.
• The training workflow takes input (datasets provided by the data loader) and passes it through a preprocessing pipeline. Processed data is fed into the model (loaded and configured by the model module), and then the training engine in the “train” subpackage handles callbacks and workflow execution.
• During training or inference, the model interacts with external monitoring/logging systems for real‑time feedback.
• The API service consumes the trained model for inference tasks, exposing endpoints that follow an OpenAI-style protocol (e.g., chat completions). The web UI is a front-end layer built on top of this API.
• Post-training, the merging/export functionality prepares fine-tuned checkpoints for deployment.
• Docker configuration files and scripts (found in the “docker” directory) abstract deployment details, ensuring that the same code base can run in various environments (CUDA, NPU, ROCm).
Architectural patterns and technologies
• Modular design and layered architecture: Each module (data, model, training, API, evaluation) is separate and can be maintained or modified independently.
• Config-driven architecture: YAML files (as seen in “examples”) are used to switch between different training modes and model configurations.
• Plugin or extensibility approach: The wide variety of supported training techniques (LoRA, QLoRA, PPO, etc.) indicates that the system is built to extend with additional modules.
• Technology stack: Python as the main programming language; dependencies include PyTorch, Hugging Face Transformers, Accelerate, PEFT, and TRL. For deployment, Docker is used and integration with external logging platforms (WandB, SwanLab) is provided.
• Emphasis on deployment across various hardware platforms (GPU, NPU, ROCm), which is managed via different Docker configurations.
◦ CLI / Command Runner
◦ Configuration Manager (YAML settings)
◦ Data Processing Pipeline (Data Loader, Converter, Formatter)
◦ Model Management Module (Model Loader, Adapter/Patcher, Quantization utilities)
◦ Training Engine (Training Workflows for SFT, DPO, PPO, etc.)
◦ Evaluation & Metrics (Evaluator, Testing Framework)
◦ API Server (Chat, inference endpoints)
◦ Web UI (Gradio-based interactive interface)
◦ External Infrastructure (Docker containers, third-party integrations like WandB, SwanLab)
◦ From the CLI passing configuration to Data Loader and Model Management.
◦ Data flows from the Data Processing Pipeline into the Training Engine.
◦ The Model Management module interacts with the Training Engine and later with the API.
◦ Evaluation results flow back into the Training Engine for callbacks and adjustments.
◦ The API Server provides output from the Model module to external clients, and the Web UI interfaces with the API.
◦ For “interfaces” (CLI, API, Web UI) use another.
◦ For “deployment infrastructure” (Docker, external integration) use a third.
• Annotations about third-party integrations (e.g., “WandB logging”, “Accelerate for multi-GPU support”, “Docker for deployment”).