# its-shashanky/lazyllm architecture

> Architecture diagram of the GitHub repository its-shashanky/lazyllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/its-shashanky/lazyllm
- Repository: https://github.com/its-shashanky/lazyllm

## Overview

##### Project type and purpose
- LazyLLM is a Python-based low-code framework for building, deploying, and iterating multi-agent AI/LLM applications.
- It exposes both a CLI (`lazyllm run …`) and a Python API (`import lazyllm → lazyllm.OnlineChatModule()`, `pipeline()`, etc.).
- It provides components (small tasks), modules (composed capabilities), flows (data orchestration), and launchers (local vs. remote execution).

##### File structure
- Top-level:
- `lazyllm/cli` → CLI entrypoints (`run`, `deploy`, `install`)
- `lazyllm/common` → shared utilities, registry, logging, threading, multiprocessing, config
- `lazyllm/components` → reusable pieces (deploy, finetune, formatter, prompter, utils)
- `lazyllm/engine` → core Flow engine, execution graph, scheduling hooks
- `lazyllm/flow` → building blocks for orchestrating data (Pipeline, Parallel, etc.)
- `lazyllm/module` → high-level modules (TrainableModule, ServerModule, WebModule, OnlineChatModule, etc.)
- `lazyllm/tools` → tool integrations (RAG, HTTP, SQL, inference servers, agents…)
- Examples illustrate usage; `docs/` holds mkdocs-based API docs and architecture image.
- CI/CD defined in `.github/workflows`, `Makefile`, `pyproject.toml`.

##### README for architecture insights
- Key patterns:
- Component-Module-Flow layering
- Registration/Discovery via `lazyllm.common.registry`
- Launcher abstraction for Local vs. Remote compute
- Data-centric Flows (pipeline/parallel/diverter)

a. Entry points
- CLI (`lazyllm run`, `lazyllm deploy`, etc.)
- Python API imports (`lazyllm.OnlineChatModule`, `pipeline`, etc.)
b. Core framework
- Module Layer:
- ActionModule, UrlModule, ServerModule, TrainableModule, WebModule, OnlineChatModule, OnlineEmbeddingModule
- Flow Layer:
- Pipeline, Parallel, Diverter, Warp, If, Loop
- Component Layer:
- Deploy components (LMDeploy, Mindie, LightLLM, vLLM, Ray…)
- Finetune components (Collie, Alpaca-LoRA, PEFT)
- Formatter, Prompter, RAG components (Retriever, Reranker, DocumentStore)
- Engine & Launcher:
- Engine builds execution graph, dispatches to Launchers
- Launchers: EmptyLauncher (local), RemoteLauncher (Slurm, SenseCore, clouds)
- Common utilities: registry, config, logging, queue, threading, redis client
c. External services & storage
- Online LLM/Evaluation services: OpenAI, SenseNova, Kimi, Zhipu, Qianwen
- Local inference frameworks: lightllm, vllm
- Fine-tune frameworks: collie, peft
- RAG storage: Vector DBs (Milvus, Chroma), Hybrid stores (Elasticsearch/OpenSearch), SQL/Mongo managers
- Document file readers/parsers under `lazyllm/tools/rag/readers`
d. Web interface
- WebModule and ServerModule spin up HTTP APIs or chat UIs
- Static assets served by `docs/assets`, optionally deployed via ReadTheDocs/GitHub Pages

##### Architectural patterns & principles
- Modular & extensible: clear boundaries between Components, Modules, and Flows
- Registration & discovery: plugins auto-registered via decorators in `common/registry`
- Launcher abstraction: decouple execution environment from business logic
- Data-driven orchestration: explicit Flows define data paths, support async/parallel
- Uniform prompts: consistent interface for online vs. local models

##### Essential elements to capture
- Entry points (CLI / Python API)
- Core engine (Flow + Launcher + Registry)
- Module layer abstraction
- Component plugins (deploy, finetune, promper, RAG tools)
- Data stores & external LLM/inference services
- Web/HTTP interface for end-user interaction
- Deployment pipelines (Docker/K8s vs. local vs. remote cluster)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry Points
    subgraph "Entry Points"
        CLI["CLI: lazyllm run/deploy/install"]:::external
        API["Python API: import lazyllm"]:::external
    end

    %% Core Framework
    subgraph "Core Framework"
        Utils["Utils"]:::core
        Registry["Registry"]:::core
        Logger["Logger"]:::core
        Engine["Engine"]:::core
        Graph["ExecutionGraph"]:::core
        Launcher["Launcher Abstraction"]:::core
    end

    %% Module Layer
    subgraph "Module Layer"
        TrainMod["TrainableModule"]:::module
        ChatMod["OnlineChatModule"]:::module
        EmbeddingMod["OnlineEmbeddingModule"]:::module
        ServerMod["ServerModule"]:::module
    end

    %% Flow Layer
    subgraph "Flow Layer"
        Pipeline["Pipeline"]:::flow
        Parallel["Parallel"]:::flow
        Diverter["Diverter"]:::flow
        Warp["Warp"]:::flow
        If["If"]:::flow
        Loop["Loop"]:::flow
    end

    %% Component Layer
    subgraph "Component Layer"
        DeployComps["Deploy Components"]:::component
        FinetuneComps["Finetune Components"]:::component
        Formatter["Formatter Components"]:::component
        Prompter["Prompter Components"]:::component
    end

    %% RAG Sub-system
    subgraph "RAG Sub-system"
        Readers["Document Readers"]:::component
        VectorStore["Vector Stores"]:::storage
        HybridStore["Hybrid Stores"]:::storage
    end

    %% External Services & Storage
    subgraph "External Services & Storage"
        OpenAI["OpenAI"]:::external
        SenseNova["SenseNova"]:::external
        Kimi["Kimi"]:::external
        Qwen["Qwen"]:::external
        LightLLM["lightllm"]:::external
        vLLM["vllm"]:::external
        Collie["collie"]:::external
        PEFT["peft"]:::external
        Milvus["Milvus"]:::storage
        Chroma["Chroma"]:::storage
        Redis["Redis"]:::storage
        MongoDB["MongoDB"]:::storage
    end

    %% Web Interface
    subgraph "Web Interface"
        WebMod["WebModule"]:::module
    end

    %% Deployment
    subgraph "Deployment"
        Local["EmptyLauncher (Local)"]:::external
        Remote["RemoteLauncher (Slurm/K8s)"]:::external
    end

    %% Connections
    CLI -->|invokes| Engine
    API -->|calls| Pipeline
    Engine -->|builds graph| Graph
    Graph -->|dispatch| Launcher
    Launcher -->|executes| Local
    Launcher -->|executes| Remote

    Registry -.-> ModuleLayer["Module Layer"] 
    Registry -.-> ComponentLayer["Component Layer"] 

    Pipeline -->|orchestrates| TrainMod
    Pipeline -->|orchestrates| ChatMod
    Pipeline -->|orchestrates| EmbeddingMod
    Pipeline -->|orchestrates| ServerMod

    Pipeline -->|sequence| DeployComps
    Parallel -->|parallel tasks| FinetuneComps
    Diverter -->|branch| Formatter
    Warp -->|wrap| Prompter

    Readers -->|ingest docs| VectorStore
    Readers -->|ingest docs| HybridStore
    VectorStore -->|lookup embeddings| ChatMod
    HybridStore -->|lookup embeddings| ChatMod

    OpenAI -->|LLM API| ChatMod
    SenseNova -->|LLM API| ChatMod
    Kimi -->|LLM API| ChatMod
    Qwen -->|LLM API| EmbeddingMod
    LightLLM -->|local inference| ChatMod
    vLLM -->|local inference| ChatMod
    Collie -->|finetune| TrainMod
    PEFT -->|finetune| TrainMod

    WebMod -->|HTTP UI| ServerMod

    Local -->|demo| WebMod
    Remote -->|production| WebMod

    %% Click Events
    click CLI "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/cli/run.py"
    click API "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/lazyml.py"
    click Utils "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/common/utils.py"
    click Registry "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/common/registry.py"
    click Logger "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/common/logger.py"
    click Engine "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/engine/engine.py"
    click Graph "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/engine/node.py"
    click Launcher "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/launcher.py"
    click TrainMod "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/module/llms/trainablemodule.py"
    click ChatMod "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/module/llms/onlinemodule/chat.py"
    click EmbeddingMod "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/module/llms/onlinemodule/embedding.py"
    click ServerMod "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/module/servermodule.py"
    click Pipeline "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/flow/flow.py"
    click DeployComps "https://github.com/its-shashanky/lazyllm/tree/main/lazyllm/components/deploy/"
    click FinetuneComps "https://github.com/its-shashanky/lazyllm/tree/main/lazyllm/components/finetune/"
    click Formatter "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/components/formatter/jsonformatter.py"
    click Prompter "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/components/prompter/prompter.py"
    click Readers "https://github.com/its-shashanky/lazyllm/tree/main/lazyllm/tools/rag/readers/"
    click VectorStore "https://github.com/its-shashanky/lazyllm/tree/main/lazyllm/tools/rag/store/vector/"
    click HybridStore "https://github.com/its-shashanky/lazyllm/tree/main/lazyllm/tools/rag/store/hybrid/"
    click WebMod "https://github.com/its-shashanky/lazyllm/blob/main/lazyllm/tools/webpages/webmodule.py"

    %% Styles
    classDef module fill:#E0F7FA,stroke:#006064
    classDef flow fill:#E1F5FE,stroke:#0D47A1
    classDef component fill:#E8F5E9,stroke:#1B5E20
    classDef storage fill:#F3E5F5,stroke:#4A148C
    classDef external fill:#FFF3E0,stroke:#E65100
    classDef core fill:#ECEFF1,stroke:#37474F
```
