# tianlan-ltd/magic-insight architecture

> Architecture diagram of the GitHub repository tianlan-ltd/magic-insight, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tianlan-ltd/magic-insight
- Repository: https://github.com/tianlan-ltd/magic-insight

## Overview

##### Project type and purpose
- Magic-BI is a full-stack, open-source ChatBI system for SQL databases.
- It exposes a web GUI (built with Svelte) and a RESTful API, coordinates LLM agents (RAG or fine-tuned), handles data retrieval/training, and manages persistence (SQL, vector DBs, object storage).

##### File structure
- Top-level directories:
- magic-frontend/ – Svelte UI components and build scripts
- magic_bi/ – Python backend modules: agents, app/core, config, data, db adapters, I/O, messaging, plugins, prompt, recommenders, training, web routers, work flows
- config/, deployment/ – YAML configs, Dockerfiles, docker-compose for services
- docs/, scripts/, setup.py, pyproject.toml

##### README insights
- Two operating modes: Agent Mode (general LLM + RAG) and Fine-Tuned Mode (custom model + agent)
- Deploys via Docker or pip, runs on GPU servers, supports private data access without requiring data warehouse
- Core features: automated data-to-SQL, LLM orchestration, model fine-tuning, and web interface

a. Main components
- Client Layer: Web UI (magic-frontend) and third-party systems calling the REST API
- API Gateway / Web Server: HTTP endpoints (magic_bi.web.* routers) and WebSocket I/O
- Core Application:
– App Manager (magic_bi.app.app_manager, app_api)
– Agent Manager & Types (magic_bi.agent) orchestrating tasks (RAG, SQL, workflow)
– Data Manager & Sources (magic_bi.data) retrieving from SQL DB or external knowledge sources
– Workflows (magic_bi.work) for document processing, summarization, search
- Model Layer:
– LLM Adapters (OpenAI, Transformers, Qwen, embeddings, reranker)
– Prompt templates and fine-tuning pipeline (magic_bi.prompt, magic_bi.train)
- Persistence & Services:
– Relational DB (PostgreSQL via SQL-ORM)
– Vector DB (Qdrant), Search Engine (Elasticsearch), Graph DB (Neo4j), Time-series (Timescale)
– Object Storage (MinIO), Message Queue (RabbitMQ)
- Training Subsystem: Offline/Online training runners (offline_main.py, train_manager)
- Deployment/Infrastructure: Docker Compose for core app, dependencies, and BI components

b. Relationships & interactions
- UI ↔ API Gateway: HTTP/ WebSocket calls
- API Gateway → Core App: Routes invoke managers (app, agent, data)
- Core App → Model Layer: Calls adapters for inference or fine-tuning
- Core App → Persistence: Reads/writes via ORM and adapters (SQL, Qdrant, Elasticsearch, MinIO)
- Core App ↔ MQ: Publishes tasks, consumes results for async workflows
- Training Subsystem → Model Layer & Persistence: Reads training data, writes fine-tuned models

c. Architectural patterns & principles
- Layered architecture: presentation → API → core domain → infrastructure
- Modular agents/ plugins: each agent type implements a strategy pattern
- Dependency injection via config files (YAML) for adapters and services
- Asynchronous workflows via message queues
- Two operating modes abstracted behind a unified agent interface

d. Key technologies
- Frontend: Svelte, Rollup
- Backend: Python 3, FastAPI/Flask-style routing (web routers), asyncio, WebSocket
- LLM frameworks: OpenAI API, Hugging Face Transformers, embeddings API
- Databases: PostgreSQL, Qdrant, Elasticsearch, Neo4j, Timescale
- Messaging: RabbitMQ
- Storage: MinIO
- Deployment: Docker, Docker-Compose

##### Diagramming conventions
- Use rectangles for components/modules (e.g., UI, Agent Manager, Data Manager)
- Use parallelograms for external services (LLM providers, external APIs)
- Presentation (blue)
- Application/API (green)
- Domain/Agent (orange)
- Data/Storage (purple)
- Integration/External (gray)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        UI["Web UI"]:::presentation
        ThirdParty["Third-party Clients"]:::presentation
    end

    %% API Layer
    subgraph "API Layer"
        APIGateway["API Gateway / Web Server"]:::api
        WS["WebSocket I/O"]:::api
        subgraph "Routers"
            AgentRouter["Agent Router"]:::api
            AppRouter["App Router"]:::api
            DataRouter["Data Router"]:::api
            DataSourceRouter["DataSource Router"]:::api
            DatasetRouter["Dataset Router"]:::api
            ModelRouter["Model Router"]:::api
            RecommendRouter["Recommend Router"]:::api
            TrainDataRouter["TrainData Router"]:::api
            TrainQaFileRouter["TrainQAFile Router"]:::api
            TrainRouter["Train Router"]:::api
            UserRouter["User Router"]:::api
        end
    end

    %% Core Layer
    subgraph "Core Layer"
        AppMgr["App Manager"]:::core
        AgentMgr["Agent Manager"]:::core
        DataMgr["Data Manager"]:::core
        Workflow["Workflows"]:::core
    end

    %% Model Layer
    subgraph "Model Layer"
        Adapters["LLM & Embedding Adapters"]:::model
        Prompts["Prompt Templates"]:::model
        Training["Training Pipeline"]:::model
    end

    %% Persistence & Services
    subgraph "Persistence Layer"
        Postgres[(PostgreSQL)]:::data
        Qdrant[(Qdrant)]:::data
        ES[(Elasticsearch)]:::data
        Neo4j[(Neo4j)]:::data
        Timescale[(Timescale)]:::data
        MinIO[(MinIO Object Storage)]:::data
        RabbitMQ[(RabbitMQ MQ)]:::data
    end

    %% External Services
    subgraph "External Services"
        LLMs["External LLM Providers"]:::external
    end

    %% Connections
    UI -->|HTTP| APIGateway
    ThirdParty -->|REST API| APIGateway
    UI -->|WebSocket| WS
    APIGateway -->|routes to| AgentRouter
    APIGateway -->|routes to| AppRouter
    APIGateway -->|routes to| DataRouter
    APIGateway -->|routes to| DataSourceRouter
    APIGateway -->|routes to| DatasetRouter
    APIGateway -->|routes to| ModelRouter
    APIGateway -->|routes to| RecommendRouter
    APIGateway -->|routes to| TrainDataRouter
    APIGateway -->|routes to| TrainQaFileRouter
    APIGateway -->|routes to| TrainRouter
    APIGateway -->|routes to| UserRouter
    WS -->|invokes| AgentRouter
    WS -->|invokes| AppRouter

    AgentRouter -->|"invoke"| AppMgr
    AppRouter -->|"invoke"| AppMgr
    DataRouter -->|"invoke"| DataMgr
    DataSourceRouter -->|"invoke"| DataMgr
    DatasetRouter -->|"invoke"| DataMgr
    ModelRouter -->|"invoke"| AgentMgr
    RecommendRouter -->|"invoke"| AgentMgr
    TrainDataRouter -->|"invoke"| Training
    TrainQaFileRouter -->|"invoke"| Training
    TrainRouter -->|"invoke"| Training
    UserRouter -->|"invoke"| AppMgr

    AppMgr -->|"manages"| AgentMgr
    AppMgr -->|"uses"| DataMgr
    AgentMgr -->|"orchestrates"| Workflow
    AgentMgr -->|"calls"| Adapters
    DataMgr -->|SQL| Postgres
    DataMgr -->|vector| Qdrant
    DataMgr -->|search| ES
    DataMgr -->|graph| Neo4j
    DataMgr -->|time-series| Timescale
    Workflow -->|async| RabbitMQ
    RabbitMQ -->|consume| Workflow
    Adapters -->|"inference"| LLMs
    Prompts -->|"templates for"| Adapters
    Training -->|"reads/writes"| Prompts
    Training -->|persist| Postgres
    Training -->|vector store| Qdrant
    Training -->|object store| MinIO

    %% Click Events
    click UI "https://github.com/tianlan-ltd/magic-insight/blob/main/magic-frontend/src/App.svelte"
    click APIGateway "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/main.py"
    click WS "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/io/websocket_io.py"
    click AgentRouter "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/web/agent_router.py"
    click AppRouter "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/web/app_router.py"
    click DataRouter "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/web/data_router.py"
    click DataSourceRouter "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/web/data_source_router.py"
    click DatasetRouter "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/web/dataset_router.py"
    click ModelRouter "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/web/model_router.py"
    click RecommendRouter "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/web/recommend_router.py"
    click TrainDataRouter "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/web/train_data_router.py"
    click TrainQaFileRouter "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/web/train_qa_file_router.py"
    click TrainRouter "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/web/train_router.py"
    click UserRouter "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/web/user_router.py"
    click AppMgr "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/app/app_manager.py"
    click AgentMgr "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/agent/agent_manager.py"
    click DataMgr "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/data/data_manager.py"
    click Workflow "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/work/work_flow.py"
    click Adapters "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/model/openai_adapter.py"
    click Prompts "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/prompt/finetuned_llm_sql_agent_prompt.py"
    click Training "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/train/train_manager.py"
    click Postgres "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/db/sql_orm.py"
    click Qdrant "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/db/qdrant_adapter.py"
    click ES "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/db/elasticsearch_adapter.py"
    click Neo4j "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/db/neo4j_adapter.py"
    click Timescale "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/db/timescale_orm.py"
    click MinIO "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/oss/minio_adapter.py"
    click RabbitMQ "https://github.com/tianlan-ltd/magic-insight/blob/main/magic_bi/mq/rabbitmq_producer.py"

    %% Styles
    classDef presentation fill:#DCEEFB,stroke:#0366D6
    classDef api fill:#DCFCE7,stroke:#15803D
    classDef core fill:#FED7AA,stroke:#C2410C
    classDef model fill:#E0E7FF,stroke:#4338CA
    classDef data fill:#F3E8FF,stroke:#6B21A8
    classDef external fill:#E5E7EB,stroke:#6B7280
```
