# jlin1201/llama-factory architecture

> Architecture diagram of the GitHub repository jlin1201/llama-factory, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jlin1201/llama-factory
- Repository: https://github.com/jlin1201/llama-factory

## Overview

##### Main components and modules
- CLI and Command Line Interface – the entry point for training, evaluation, chat, exporting/merging models, etc. (located in the “src/llamafactory/cli.py” and top-level “train.py”).
- API Service – an OpenAI-style API server for inference (e.g., “src/api.py” and “src/llamafactory/api/…”). This module handles endpoints (chat, image testing, toolcall, etc.) and is responsible for interfacing with external applications.
- Web UI – a Gradio-powered GUI (under “src/llamafactory/webui”) that provides an interactive interface for fine-tuning and model inference.
- Training Subsystem – comprising multiple training workflows and fine-tuning strategies organized in “src/llamafactory/train”. Here you find dedicated submodules (for SFT, DPO, PPO, KTO, etc.) along with callback and workflow scripts.
- Data Processing and Loader – the “src/llamafactory/data” package that handles dataset loading, conversion, formatting, and collating. It supports both local data as well as integration with datasets on hubs.
- Model and Factory Layers – found in “src/llamafactory/model”, “src/llamafactory/llamafactory” (which includes modules for loading, patching, adapter layers, quantization, etc.). These components abstract over various LLMs and support adapters like LoRA, quantized training modules, and other model utilities.
- Evaluation Module – under “src/llamafactory/eval”, which defines evaluators, metrics, and provides interfaces to test aspects of fine-tuning.
- External Integrations and Extras – additional utilities (logging, hyperparameter parsing, misc constants, and extra algorithms in “src/llamafactory/extras” and “src/llamafactory/hparams”) that enable integrations with logging platforms (WandB, SwanLab), Docker deployments, and even specialized optimizers (GaLore, Adam-mini, etc.).

##### Relationships and data flows
- The CLI acts as an orchestrator and selects a YAML configuration that specifies a chosen training routine or inference setup.
- The training workflow takes input (datasets provided by the data loader) and passes it through a preprocessing pipeline. Processed data is fed into the model (loaded and configured by the model module), and then the training engine in the “train” subpackage handles callbacks and workflow execution.
- During training or inference, the model interacts with external monitoring/logging systems for real‑time feedback.
- The API service consumes the trained model for inference tasks, exposing endpoints that follow an OpenAI-style protocol (e.g., chat completions). The web UI is a front-end layer built on top of this API.
- Post-training, the merging/export functionality prepares fine-tuned checkpoints for deployment.
- Docker configuration files and scripts (found in the “docker” directory) abstract deployment details, ensuring that the same code base can run in various environments (CUDA, NPU, ROCm).

##### Architectural patterns and technologies
- Modular design and layered architecture: Each module (data, model, training, API, evaluation) is separate and can be maintained or modified independently.
- Config-driven architecture: YAML files (as seen in “examples”) are used to switch between different training modes and model configurations.
- Plugin or extensibility approach: The wide variety of supported training techniques (LoRA, QLoRA, PPO, etc.) indicates that the system is built to extend with additional modules.
- Technology stack: Python as the main programming language; dependencies include PyTorch, Hugging Face Transformers, Accelerate, PEFT, and TRL. For deployment, Docker is used and integration with external logging platforms (WandB, SwanLab) is provided.
- Emphasis on deployment across various hardware platforms (GPU, NPU, ROCm), which is managed via different Docker configurations.

◦ CLI / Command Runner
◦ Configuration Manager (YAML settings)
◦ Data Processing Pipeline (Data Loader, Converter, Formatter)
◦ Model Management Module (Model Loader, Adapter/Patcher, Quantization utilities)
◦ Training Engine (Training Workflows for SFT, DPO, PPO, etc.)
◦ Evaluation & Metrics (Evaluator, Testing Framework)
◦ API Server (Chat, inference endpoints)
◦ Web UI (Gradio-based interactive interface)
◦ External Infrastructure (Docker containers, third-party integrations like WandB, SwanLab)
◦ From the CLI passing configuration to Data Loader and Model Management.
◦ Data flows from the Data Processing Pipeline into the Training Engine.
◦ The Model Management module interacts with the Training Engine and later with the API.
◦ Evaluation results flow back into the Training Engine for callbacks and adjustments.
◦ The API Server provides output from the Model module to external clients, and the Web UI interfaces with the API.
◦ For “interfaces” (CLI, API, Web UI) use another.
◦ For “deployment infrastructure” (Docker, external integration) use a third.
- Annotations about third-party integrations (e.g., “WandB logging”, “Accelerate for multi-GPU support”, “Docker for deployment”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "User Input Layer"
        CLI["CLI / Command Runner"]:::interface
        Config["Configuration Manager"]:::interface
    end

    subgraph "Data & Model Layer"
        Data["Data Processing Pipeline"]:::core
        Model["Model Management Module"]:::core
    end

    subgraph "Training & Evaluation Layer"
        Training["Training Engine"]:::core
        Eval["Evaluation & Metrics"]:::core
    end

    subgraph "Inference & Interface Layer"
        API["API Service"]:::interface
        WebUI["Web UI"]:::interface
    end

    subgraph "External Infrastructure"
        External["External Integrations"]:::external
        subgraph "Deployment"
            Docker_CUDA["Docker (CUDA)"]:::external
            Docker_NPU["Docker (NPU)"]:::external
            Docker_ROCm["Docker (ROCm)"]:::external
        end
    end

    %% Data Flow Connections
    CLI -->|"selectsConfig"| Config
    Config -->|"providesSettings"| Data
    Config -->|"providesSettings"| Model
    Data -->|"DataInput"| Training
    Model -->|"ModelParams"| Training
    Training -->|"feedback"| Eval
    Eval -->|"feedback"| Training
    Training -->|"trainedModel"| API
    Model -->|"modelLoading"| API
    API -->|"inferenceReq"| WebUI
    Training -->|"export/merge"| Model

    %% External and Deployment Connections
    Training -->|"logsMonitoring"| External
    External -->|"deploysOn"| Docker_CUDA
    External -->|"deploysOn"| Docker_NPU
    External -->|"deploysOn"| Docker_ROCm

    %% Click Events
    click CLI "https://github.com/jlin1201/llama-factory/blob/main/src/llamafactory/cli.py"
    click CLI "https://github.com/jlin1201/llama-factory/blob/main/src/train.py"
    click API "https://github.com/jlin1201/llama-factory/blob/main/src/api.py"
    click API "https://github.com/jlin1201/llama-factory/tree/main/src/llamafactory/api/"
    click WebUI "https://github.com/jlin1201/llama-factory/tree/main/src/llamafactory/webui"
    click Training "https://github.com/jlin1201/llama-factory/tree/main/src/llamafactory/train"
    click Data "https://github.com/jlin1201/llama-factory/tree/main/src/llamafactory/data"
    click Model "https://github.com/jlin1201/llama-factory/tree/main/src/llamafactory/model"
    click Eval "https://github.com/jlin1201/llama-factory/tree/main/src/llamafactory/eval"
    click External "https://github.com/jlin1201/llama-factory/tree/main/src/llamafactory/extras"
    click External "https://github.com/jlin1201/llama-factory/tree/main/src/llamafactory/hparams"
    click Docker_CUDA "https://github.com/jlin1201/llama-factory/tree/main/docker/docker-cuda"
    click Docker_NPU "https://github.com/jlin1201/llama-factory/tree/main/docker/docker-npu"
    click Docker_ROCm "https://github.com/jlin1201/llama-factory/tree/main/docker/docker-rocm"

    %% Styles
    classDef interface fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef core fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef external fill:#FFD580,stroke:#000,stroke-width:2px;
```
