How hiyouga/llama-factory works
Project Type & Purpose
• It’s a fine‐tuning platform tailored for large language model training, inference, and evaluation.
• Its main goals include zero-code CLI training, API-based deployment, web UI for interactive use, Docker-based containerization, and integration of advanced methods like LoRA, QLoRA, reward modeling, and more.
• The project incorporates a rich set of examples, training pipelines, data processing utilities, evaluation and logging modules, and support for multiple hardware backends (CUDA, NPU, ROCm).
Main Components
– Core Engine (src/llamafactory): This contains the bulk of the business logic. It is split into many sub-modules:
• API & Chat Modules (e.g., src/llamafactory/api, src/llamafactory/chat): Handle OpenAI-style API requests, chat interactions, and protocol/API alignment.
• Data Pipelines (src/llamafactory/data): Contains collators, converters, loaders, processors, and templates for preparing and transforming datasets.
• Training Modules (src/llamafactory/train): Contains workflows, trainers (for DPO, PPO, KTO, supervised fine-tuning, etc.), callbacks, process management, and tuning logic.
• Model Utilities (src/llamafactory/model and submodules): Functions for loading, patching, quantization, adapter integrations, and other model operations.
• Parameter & Hyper-parameter Management (src/llamafactory/hparams): Houses configuration parsing and definitions for different training/inference setups.
• Evaluation Modules (src/llamafactory/eval): Components to run evaluations including standard metrics and templates.
• Extras (src/llamafactory/extras): Contains logging, environment management, and support for additional algorithms (like GaLore, BAdam, etc.).
• Web UI (src/llamafactory/webui): Implements the Gradio-powered interactive interface for training, evaluation, and inference.
– CLI and Scripting Interfaces:
• The cli entry-points (src/llamafactory/cli.py and scripts folder) provide command-line control for training, API launching, merging and export duties.
– Docker & Deployment:
• The docker folders (docker/docker-cuda, docker/docker-npu, docker/docker-rocm) define different container environments.
• They show how the project packages dependencies and runtime settings (with docker-compose, Dockerfiles).
– Examples & Tests:
• The examples directory shows YAML configuration examples for training, fine-tuning, inference, and merging.
• The tests directory ensures code quality via unit tests for different modules.
– Data & Assets:
• The data folder contains demo datasets, dataset information, and sample JSON/other media files that illustrate supported tasks.
• The assets folder provides visual elements (logos, benchmark images) and supplementary user guidance.
Relationships and Interactions
• How the CLI/Web UI interacts with the Core Engine:
→ From user input (via CLI commands or web interface) to configuration parsing (hparams) and then delegation to the training pipeline.
• Data flow from the Data Pipeline module to the Training Engine and Evaluation components.
• API handlers (under api and chat) with external clients (for deployment and inference), either through OpenAI-style REST endpoints or via vLLM integration.
• Docker/environment layer as encompassing the entire system for deployment in various hardware setups.
Architectural Patterns & Technologies
– The design follows a modular architecture where each sub-system (data, training, evaluation, API, web UI) is isolated but interacts via well-defined protocols.
– Separation of concerns is evident: configuration in YAML examples, business logic in src, and infrastructural code in docker.
– Technologies include Python (with packages like torch, transformers, accelerate, PEFT, TRL), containerization (Docker), and interactive UI frameworks (Gradio).
– There is also a test-driven structure (with tests directory) that reflects an emphasis on robustness.