How tianlan-ltd/magic-insight works
Project type and purpose
• Magic-BI is a full-stack, open-source ChatBI system for SQL databases.
• It exposes a web GUI (built with Svelte) and a RESTful API, coordinates LLM agents (RAG or fine-tuned), handles data retrieval/training, and manages persistence (SQL, vector DBs, object storage).
File structure
• Top-level directories:
• magic-frontend/ – Svelte UI components and build scripts
• magic_bi/ – Python backend modules: agents, app/core, config, data, db adapters, I/O, messaging, plugins, prompt, recommenders, training, web routers, work flows
• config/, deployment/ – YAML configs, Dockerfiles, docker-compose for services
• docs/, scripts/, setup.py, pyproject.toml
README insights
• Two operating modes: Agent Mode (general LLM + RAG) and Fine-Tuned Mode (custom model + agent)
• Deploys via Docker or pip, runs on GPU servers, supports private data access without requiring data warehouse
• Core features: automated data-to-SQL, LLM orchestration, model fine-tuning, and web interface
a. Main components
• Client Layer: Web UI (magic-frontend) and third-party systems calling the REST API
• API Gateway / Web Server: HTTP endpoints (magic_bi.web.* routers) and WebSocket I/O
• Core Application:
– App Manager (magic_bi.app.app_manager, app_api)
– Agent Manager & Types (magic_bi.agent) orchestrating tasks (RAG, SQL, workflow)
– Data Manager & Sources (magic_bi.data) retrieving from SQL DB or external knowledge sources
– Workflows (magic_bi.work) for document processing, summarization, search
• Model Layer:
– LLM Adapters (OpenAI, Transformers, Qwen, embeddings, reranker)
– Prompt templates and fine-tuning pipeline (magic_bi.prompt, magic_bi.train)
• Persistence & Services:
– Relational DB (PostgreSQL via SQL-ORM)
– Vector DB (Qdrant), Search Engine (Elasticsearch), Graph DB (Neo4j), Time-series (Timescale)
– Object Storage (MinIO), Message Queue (RabbitMQ)
• Training Subsystem: Offline/Online training runners (offline_main.py, train_manager)
• Deployment/Infrastructure: Docker Compose for core app, dependencies, and BI components
b. Relationships & interactions
• UI ↔ API Gateway: HTTP/ WebSocket calls
• API Gateway → Core App: Routes invoke managers (app, agent, data)
• Core App → Model Layer: Calls adapters for inference or fine-tuning
• Core App → Persistence: Reads/writes via ORM and adapters (SQL, Qdrant, Elasticsearch, MinIO)
• Core App ↔ MQ: Publishes tasks, consumes results for async workflows
• Training Subsystem → Model Layer & Persistence: Reads training data, writes fine-tuned models
c. Architectural patterns & principles
• Layered architecture: presentation → API → core domain → infrastructure
• Modular agents/ plugins: each agent type implements a strategy pattern
• Dependency injection via config files (YAML) for adapters and services
• Asynchronous workflows via message queues
• Two operating modes abstracted behind a unified agent interface
d. Key technologies
• Frontend: Svelte, Rollup
• Backend: Python 3, FastAPI/Flask-style routing (web routers), asyncio, WebSocket
• LLM frameworks: OpenAI API, Hugging Face Transformers, embeddings API
• Databases: PostgreSQL, Qdrant, Elasticsearch, Neo4j, Timescale
• Messaging: RabbitMQ
• Storage: MinIO
• Deployment: Docker, Docker-Compose
Diagramming conventions
• Use rectangles for components/modules (e.g., UI, Agent Manager, Data Manager)
• Use parallelograms for external services (LLM providers, external APIs)
• Presentation (blue)
• Application/API (green)
• Domain/Agent (orange)
• Data/Storage (purple)
• Integration/External (gray)