How givgramacho/llama-factory works
Main Components
• Core Library (src/llamafactory): This is the heart of the system and includes sub-modules:
– API: (src/llamafactory/api and src/api.py) providing chat and OpenAI-style endpoints.
– Chat & Inference Engines: (src/llamafactory/chat) which handles various inference backends (e.g., HF engine, vLLM engine).
– Data Processing and Loader: (src/llamafactory/data) that manages dataset ingestion, collation, conversion, formatting, and template handling.
– Evaluation Modules: (src/llamafactory/eval) that incorporate different evaluation strategies.
– Training Components: (src/llamafactory/train) with submodules for each training approach (SFT, PPO, DPO, KTO, etc.) and additional utilities (callbacks, workflow management).
– Model Components: (src/llamafactory/model) responsible for loading, patching, and handling operations like adapter injection, quantization, and merging.
– Hyperparameters and Configuration: (src/llamafactory/hparams) to parse and manage training/evaluation configurations.
– Extra Utilities: (src/llamafactory/extras) for logging, environment configuration, and miscellaneous helper functions.
– Web UI: (src/llamafactory/webui) to support the integrated GUI (powered by Gradio).
• CLI & Scripts:
– The CLI interface (e.g., src/llamafactory/cli.py) which acts as a gateway to all functionalities (training, inference, export).
– Additional scripts (in the “scripts” folder) for use cases like checkpoint conversion, inference with vLLM, tool call examples, and statistical utilities.
• Data & Datasets:
– The “data” directory and its README – holds datasets, demo files, metadata, and additional custom examples.
• Testing & Evaluation:
– The “tests” directory indicates unit tests for data processors, model utilities, training workflows, and end-to-end training and chat/inference pipelines.
• Deployment & Containerization:
– Docker files under the “docker” directory show how containers for different compute environments (CUDA, NPU, ROCm) are built, with corresponding docker-compose configurations.
• Examples & Configuration Files:
– The “examples” directory includes many YAML configuration files that demonstrate various fine-tuning, inference, and merging scenarios.
• Documentation & Project Metadata:
– README files (both English and Chinese), license, citation info, and support for external links (documentation, colab, etc.)
Relationships and Interactions
• Data Flow:
– Datasets in the “data” folder feed into data processing modules (collator, parser) in the core library.
– Processed data is used by the training modules.
• Model Workflow:
– Training modules are fed configuration parameters (from hparams and YAML examples) to train, fine-tune, and evaluate models.
– Trained models are loaded and processed by the model module for inference via the API or Web UI.
• Interface and API:
– CLI calls wrap around the core functionalities and abstract complexities for users.
– The web UI (in webui) is integrated as an additional layer client for interactive fine-tuning/inference.
– API endpoints (in src/api.py and api submodule) expose an OpenAI-style interface for chat and inference.
• Deployment:
– Docker containers encapsulate the entire environment for different hardware (CUDA, NPU, ROCm) ensuring portability.
– Volume mappings (e.g., for caching and data) link the host to the containerized system.
• Testing & Evaluation:
– The “tests” folder ensures unit testing and integration testing of most components. These tests capture the expected behavior of data processing, model loading, and training pipelines.
Important Architectural Patterns and Design Principles
• Modular Design: Components are separated into modules (data, model, train, eval, webui) that can be maintained almost independently. This is visible in the folder structure.
• Configurability: Multiple YAML configurations (under examples/) and a flexible hyperparameter module (hparams) allow for custom training/inference setups.
• Extensibility: The project supports a wide range of model types, training paradigms, and deployments through plugins and adapter modules.
• Containerization: Docker support is provided for reproducible containerized environments on different hardware platforms.
• CLI-First Approach: A command line interface encapsulates many workflows, making it easier to launch training, inference, export, and API services.
Relevant Technologies and Frameworks
• Python is the primary language.
• Key libraries include torch (PyTorch), transformers, datasets, accelerate, and PEFT.
• Deployment integrations with Docker, docker-compose.
• Web UI is powered by Gradio.
• Supporting tools for quantization (bitsandbytes, QLoRA), distributed training (deepspeed), and fast inference (vLLM).
▸ Data & Preprocessing Layer – representing “data”, “data/processor”, “data/collator”, etc.
▸ Model Training & Evaluation Layer – including modules from “train”, “model”, “eval”, “hparams” and configuration files in “examples”.
▸ Inference & API Interaction Layer – showing components from “api”, “chat”, “webui”, and “cli”.
▸ Deployment & Containerization Layer – representing Docker setups in “docker”, and the links to hardware with CUDA/NPU/ROCm.
▸ Testing & Tools Layer – referring to the “tests” folder and “scripts” for conversion and utility tasks.
• Annotations to indicate the interactions (e.g., “data flows from dataset files to collator → trainer”, “trained models are served via API and Web UI”, “CLI orchestrates these pipelines”, “Docker containers encapsulate runtime dependencies”).