# givgramacho/llama-factory architecture

> Architecture diagram of the GitHub repository givgramacho/llama-factory, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/givgramacho/llama-factory
- Repository: https://github.com/givgramacho/llama-factory

## Overview

##### Main Components
- Core Library (src/llamafactory): This is the heart of the system and includes sub-modules:
– API: (src/llamafactory/api and src/api.py) providing chat and OpenAI-style endpoints.
– Chat & Inference Engines: (src/llamafactory/chat) which handles various inference backends (e.g., HF engine, vLLM engine).
– Data Processing and Loader: (src/llamafactory/data) that manages dataset ingestion, collation, conversion, formatting, and template handling.
– Evaluation Modules: (src/llamafactory/eval) that incorporate different evaluation strategies.
– Training Components: (src/llamafactory/train) with submodules for each training approach (SFT, PPO, DPO, KTO, etc.) and additional utilities (callbacks, workflow management).
– Model Components: (src/llamafactory/model) responsible for loading, patching, and handling operations like adapter injection, quantization, and merging.
– Hyperparameters and Configuration: (src/llamafactory/hparams) to parse and manage training/evaluation configurations.
– Extra Utilities: (src/llamafactory/extras) for logging, environment configuration, and miscellaneous helper functions.
– Web UI: (src/llamafactory/webui) to support the integrated GUI (powered by Gradio).
- CLI & Scripts:
– The CLI interface (e.g., src/llamafactory/cli.py) which acts as a gateway to all functionalities (training, inference, export).
– Additional scripts (in the “scripts” folder) for use cases like checkpoint conversion, inference with vLLM, tool call examples, and statistical utilities.
- Data & Datasets:
– The “data” directory and its README – holds datasets, demo files, metadata, and additional custom examples.
- Testing & Evaluation:
– The “tests” directory indicates unit tests for data processors, model utilities, training workflows, and end-to-end training and chat/inference pipelines.
- Deployment & Containerization:
– Docker files under the “docker” directory show how containers for different compute environments (CUDA, NPU, ROCm) are built, with corresponding docker-compose configurations.
- Examples & Configuration Files:
– The “examples” directory includes many YAML configuration files that demonstrate various fine-tuning, inference, and merging scenarios.
- Documentation & Project Metadata:
– README files (both English and Chinese), license, citation info, and support for external links (documentation, colab, etc.)

##### Relationships and Interactions
- Data Flow:
– Datasets in the “data” folder feed into data processing modules (collator, parser) in the core library.
– Processed data is used by the training modules.
- Model Workflow:
– Training modules are fed configuration parameters (from hparams and YAML examples) to train, fine-tune, and evaluate models.
– Trained models are loaded and processed by the model module for inference via the API or Web UI.
- Interface and API:
– CLI calls wrap around the core functionalities and abstract complexities for users.
– The web UI (in webui) is integrated as an additional layer client for interactive fine-tuning/inference.
– API endpoints (in src/api.py and api submodule) expose an OpenAI-style interface for chat and inference.
- Deployment:
– Docker containers encapsulate the entire environment for different hardware (CUDA, NPU, ROCm) ensuring portability.
– Volume mappings (e.g., for caching and data) link the host to the containerized system.
- Testing & Evaluation:
– The “tests” folder ensures unit testing and integration testing of most components. These tests capture the expected behavior of data processing, model loading, and training pipelines.

##### Important Architectural Patterns and Design Principles
- Modular Design: Components are separated into modules (data, model, train, eval, webui) that can be maintained almost independently. This is visible in the folder structure.
- Configurability: Multiple YAML configurations (under examples/) and a flexible hyperparameter module (hparams) allow for custom training/inference setups.
- Extensibility: The project supports a wide range of model types, training paradigms, and deployments through plugins and adapter modules.
- Containerization: Docker support is provided for reproducible containerized environments on different hardware platforms.
- CLI-First Approach: A command line interface encapsulates many workflows, making it easier to launch training, inference, export, and API services.

##### Relevant Technologies and Frameworks
- Python is the primary language.
- Key libraries include torch (PyTorch), transformers, datasets, accelerate, and PEFT.
- Deployment integrations with Docker, docker-compose.
- Web UI is powered by Gradio.
- Supporting tools for quantization (bitsandbytes, QLoRA), distributed training (deepspeed), and fast inference (vLLM).

▸ Data & Preprocessing Layer – representing “data”, “data/processor”, “data/collator”, etc.
▸ Model Training & Evaluation Layer – including modules from “train”, “model”, “eval”, “hparams” and configuration files in “examples”.
▸ Inference & API Interaction Layer – showing components from “api”, “chat”, “webui”, and “cli”.
▸ Deployment & Containerization Layer – representing Docker setups in “docker”, and the links to hardware with CUDA/NPU/ROCm.
▸ Testing & Tools Layer – referring to the “tests” folder and “scripts” for conversion and utility tasks.
- Annotations to indicate the interactions (e.g., “data flows from dataset files to collator → trainer”, “trained models are served via API and Web UI”, “CLI orchestrates these pipelines”, “Docker containers encapsulate runtime dependencies”).

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% Core Library Parent & Modules
    CoreLib["Core Library"]:::core
    subgraph "Core Modules"
        API["API Module"]:::core
        Chat["Chat & Inference Engines"]:::core
        DataProc["Data Processing & Loader"]:::core
        Eval["Evaluation Modules"]:::core
        Train["Training Components"]:::core
        Model["Model Components"]:::core
        Hparams["Hyperparameters & Config"]:::core
        Extras["Extra Utilities"]:::core
        WebUI["Web UI"]:::core
    end

    %% Interface and Orchestration Layer
    subgraph "Interface & Orchestration"
        CLI["CLI Interface"]:::interface
        Scripts["Additional Scripts"]:::interface
    end

    %% Data and Configuration Layer
    subgraph "Data & Configuration"
        Datasets["Data & Datasets"]:::data
        Examples["Examples & Config Files"]:::config
        Docs["Documentation & Project Metadata"]:::docs
    end

    %% Testing and Deployment Layer
    subgraph "Testing & Deployment"
        Tests["Testing & Evaluation"]:::test
        Deployment["Deployment & Containerization"]:::deploy
    end

    %% Data Flow & Interactions
    Datasets -->|"feeds"| DataProc
    Examples -->|"provides_configs_to"| Hparams
    Hparams -->|"configures"| Train
    Train -->|"produces_models_for"| Model
    Model -->|"served_by"| API
    Model -->|"served_by"| WebUI
    CLI -->|"orchestrates"| Train
    CLI -->|"triggers"| API
    Scripts -->|"assist_with"| Train
    API -->|"inference_calls"| Chat
    Tests -->|"validate"| DataProc
    Tests -->|"validate"| Train
    Tests -->|"validate"| Model
    Deployment -->|"containerizes"| CoreLib
    Docs --- Examples

    %% Core Library Composition
    CoreLib -->|"contains"| API
    CoreLib -->|"contains"| Chat
    CoreLib -->|"contains"| DataProc
    CoreLib -->|"contains"| Eval
    CoreLib -->|"contains"| Train
    CoreLib -->|"contains"| Model
    CoreLib -->|"contains"| Hparams
    CoreLib -->|"contains"| Extras
    CoreLib -->|"contains"| WebUI

    %% Click Events
    click CoreLib "https://github.com/givgramacho/llama-factory/tree/main/src/llamafactory"
    click API "https://github.com/givgramacho/llama-factory/tree/main/src/llamafactory/api"
    click Chat "https://github.com/givgramacho/llama-factory/tree/main/src/llamafactory/chat"
    click DataProc "https://github.com/givgramacho/llama-factory/tree/main/src/llamafactory/data"
    click Eval "https://github.com/givgramacho/llama-factory/tree/main/src/llamafactory/eval"
    click Train "https://github.com/givgramacho/llama-factory/tree/main/src/llamafactory/train"
    click Model "https://github.com/givgramacho/llama-factory/tree/main/src/llamafactory/model"
    click Hparams "https://github.com/givgramacho/llama-factory/tree/main/src/llamafactory/hparams"
    click Extras "https://github.com/givgramacho/llama-factory/tree/main/src/llamafactory/extras"
    click WebUI "https://github.com/givgramacho/llama-factory/tree/main/src/llamafactory/webui"
    click CLI "https://github.com/givgramacho/llama-factory/blob/main/src/llamafactory/cli.py"
    click Scripts "https://github.com/givgramacho/llama-factory/tree/main/scripts"
    click Datasets "https://github.com/givgramacho/llama-factory/tree/main/data"
    click Tests "https://github.com/givgramacho/llama-factory/tree/main/tests"
    click Deployment "https://github.com/givgramacho/llama-factory/tree/main/docker"
    click Examples "https://github.com/givgramacho/llama-factory/tree/main/examples"
    click Docs "https://github.com/givgramacho/llama-factory/blob/main/README.md"

    %% Styles
    classDef core fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef interface fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef data fill:#FFFACD,stroke:#000,stroke-width:2px;
    classDef config fill:#FFB6C1,stroke:#000,stroke-width:2px;
    classDef docs fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef test fill:#F08080,stroke:#000,stroke-width:2px;
    classDef deploy fill:#FFD700,stroke:#000,stroke-width:2px;
```
