# hiyouga/llama-factory architecture

> Architecture diagram of the GitHub repository hiyouga/llama-factory, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hiyouga/llama-factory
- Repository: https://github.com/hiyouga/llama-factory

## Overview

##### Project Type & Purpose
- It’s a fine‐tuning platform tailored for large language model training, inference, and evaluation.
- Its main goals include zero-code CLI training, API-based deployment, web UI for interactive use, Docker-based containerization, and integration of advanced methods like LoRA, QLoRA, reward modeling, and more.
- The project incorporates a rich set of examples, training pipelines, data processing utilities, evaluation and logging modules, and support for multiple hardware backends (CUDA, NPU, ROCm).

##### Main Components
– Core Engine (src/llamafactory): This contains the bulk of the business logic. It is split into many sub-modules:
- API & Chat Modules (e.g., src/llamafactory/api, src/llamafactory/chat): Handle OpenAI-style API requests, chat interactions, and protocol/API alignment.
- Data Pipelines (src/llamafactory/data): Contains collators, converters, loaders, processors, and templates for preparing and transforming datasets.
- Training Modules (src/llamafactory/train): Contains workflows, trainers (for DPO, PPO, KTO, supervised fine-tuning, etc.), callbacks, process management, and tuning logic.
- Model Utilities (src/llamafactory/model and submodules): Functions for loading, patching, quantization, adapter integrations, and other model operations.
- Parameter & Hyper-parameter Management (src/llamafactory/hparams): Houses configuration parsing and definitions for different training/inference setups.
- Evaluation Modules (src/llamafactory/eval): Components to run evaluations including standard metrics and templates.
- Extras (src/llamafactory/extras): Contains logging, environment management, and support for additional algorithms (like GaLore, BAdam, etc.).
- Web UI (src/llamafactory/webui): Implements the Gradio-powered interactive interface for training, evaluation, and inference.
– CLI and Scripting Interfaces:
- The cli entry-points (src/llamafactory/cli.py and scripts folder) provide command-line control for training, API launching, merging and export duties.
– Docker & Deployment:
- The docker folders (docker/docker-cuda, docker/docker-npu, docker/docker-rocm) define different container environments.
- They show how the project packages dependencies and runtime settings (with docker-compose, Dockerfiles).
– Examples & Tests:
- The examples directory shows YAML configuration examples for training, fine-tuning, inference, and merging.
- The tests directory ensures code quality via unit tests for different modules.
– Data & Assets:
- The data folder contains demo datasets, dataset information, and sample JSON/other media files that illustrate supported tasks.
- The assets folder provides visual elements (logos, benchmark images) and supplementary user guidance.

##### Relationships and Interactions
- How the CLI/Web UI interacts with the Core Engine:
→ From user input (via CLI commands or web interface) to configuration parsing (hparams) and then delegation to the training pipeline.
- Data flow from the Data Pipeline module to the Training Engine and Evaluation components.
- API handlers (under api and chat) with external clients (for deployment and inference), either through OpenAI-style REST endpoints or via vLLM integration.
- Docker/environment layer as encompassing the entire system for deployment in various hardware setups.

##### Architectural Patterns & Technologies
– The design follows a modular architecture where each sub-system (data, training, evaluation, API, web UI) is isolated but interacts via well-defined protocols.
– Separation of concerns is evident: configuration in YAML examples, business logic in src, and infrastructural code in docker.
– Technologies include Python (with packages like torch, transformers, accelerate, PEFT, TRL), containerization (Docker), and interactive UI frameworks (Gradio).
– There is also a test-driven structure (with tests directory) that reflects an emphasis on robustness.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interfaces
    subgraph "User Interfaces"
        CLI["CLI Entry-point"]:::ui
        WebUI["Web UI"]:::ui
    end

    %% Core Engine and its submodules
    subgraph "Core Engine"
        CoreEngine["Core Engine"]:::core
        API["API Module"]:::core
        Chat["Chat Module"]:::core
        DataP["Data Pipelines"]:::core
        Train["Training Modules"]:::core
        Model["Model Utilities"]:::core
        Hparams["Hyper-parameter Management"]:::core
        Eval["Evaluation Modules"]:::core
        Extras["Extras"]:::core
    end

    %% Infrastructure/Deployment Layer
    subgraph "Infrastructure"
        DockerCUDA["Docker CUDA"]:::infra
        DockerNPU["Docker NPU"]:::infra
        DockerROCm["Docker ROCm"]:::infra
    end

    %% Supporting Components
    subgraph "Supporting"
        Examples["Examples"]:::support
        Tests["Tests"]:::support
        DataFolder["Data"]:::support
        Assets["Assets"]:::support
    end

    %% Top-level UI interactions with Core Engine
    CLI -->|"feeds_to"| CoreEngine
    WebUI -->|"calls"| CoreEngine

    %% Core Engine internal structure
    CoreEngine --> Hparams
    CoreEngine --> API
    CoreEngine --> DataP
    CoreEngine --> Train
    CoreEngine --> Model
    CoreEngine --> Eval
    CoreEngine --> Extras

    %% Inter-module relationships within Core Engine
    Hparams -->|"configures"| Train
    DataP -->|"feeds_data"| Train
    Train -->|"uses"| Model
    Train -->|"evaluates_to"| Eval
    API -->|"handles_chat"| Chat
    Train -->|"logs_via"| Extras
    Eval -->|"logs_via"| Extras

    %% Deployment interactions
    CoreEngine -->|"deployed_with"| DockerCUDA
    CoreEngine -->|"deployed_with"| DockerNPU
    CoreEngine -->|"deployed_with"| DockerROCm

    %% Supporting integration
    Examples -->|"config_examples"| CoreEngine
    Tests -->|"testing"| CoreEngine
    DataFolder -->|"supplies_data"| CoreEngine
    Assets -->|"supplies_assets"| CoreEngine

    %% Click Events
    click CoreEngine "https://github.com/hiyouga/llama-factory/tree/main/src/llamafactory"
    click API "https://github.com/hiyouga/llama-factory/tree/main/src/llamafactory/api"
    click Chat "https://github.com/hiyouga/llama-factory/tree/main/src/llamafactory/chat"
    click DataP "https://github.com/hiyouga/llama-factory/tree/main/src/llamafactory/data"
    click Train "https://github.com/hiyouga/llama-factory/tree/main/src/llamafactory/train"
    click Model "https://github.com/hiyouga/llama-factory/tree/main/src/llamafactory/model"
    click Hparams "https://github.com/hiyouga/llama-factory/tree/main/src/llamafactory/hparams"
    click Eval "https://github.com/hiyouga/llama-factory/tree/main/src/llamafactory/eval"
    click Extras "https://github.com/hiyouga/llama-factory/tree/main/src/llamafactory/extras"
    click WebUI "https://github.com/hiyouga/llama-factory/tree/main/src/llamafactory/webui"
    click CLI "https://github.com/hiyouga/llama-factory/blob/main/src/llamafactory/cli.py"
    click DockerCUDA "https://github.com/hiyouga/llama-factory/tree/main/docker/docker-cuda"
    click DockerNPU "https://github.com/hiyouga/llama-factory/tree/main/docker/docker-npu"
    click DockerROCm "https://github.com/hiyouga/llama-factory/tree/main/docker/docker-rocm"
    click Examples "https://github.com/hiyouga/llama-factory/tree/main/examples"
    click Tests "https://github.com/hiyouga/llama-factory/tree/main/tests"
    click DataFolder "https://github.com/hiyouga/llama-factory/tree/main/data"
    click Assets "https://github.com/hiyouga/llama-factory/tree/main/assets"

    %% Styles
    classDef ui fill:#a2d2ff,stroke:#084298,stroke-width:2px;
    classDef core fill:#caffbf,stroke:#007f5f,stroke-width:2px;
    classDef infra fill:#ffe066,stroke:#e59500,stroke-width:2px;
    classDef support fill:#ffadad,stroke:#e63946,stroke-width:2px;
```
