How deepinsight-ai/deepbi works
Main Components
• Frontend (Client): All files under the “client” directory indicate a rich user interface built in JavaScript/React with assets, components, pages, and tests. This layer handles user interactions (dialogues, dashboards, query interfaces, and visualizations).
• Backend – AI and BI Layers:
• AI Components: The “ai” directory is subdivided into agents, agentchat, prompts, and backend. These modules handle conversational query generation, interfacing with large language models, and executing natural language tasks (query formulation and interactive dialogue).
• BI Components: The “bi” directory includes authentication, CLI tools, destination integration, handlers, models, query runners, tasks, and other business logic modules. This represents the business intelligence side that handles reporting, dashboard assembly, alerts, data source connectivity, and permissions.
• Data Source & Database Abstraction: Within both “ai/backend/aidb” and the “bi” modules, there are specific files dealing with database utilities, query runners, and adapters. They support multiple databases (MySQL, PostgreSQL, MongoDB, Doris, StarRocks, CSV/Excel) and represent the project’s data access layer.
• Deployment and Infrastructure: Files such as docker-compose.yml, Dockerfile.template, and various installation scripts (Install.sh, ubuntu_install.sh, etc.) indicate containerized deployment. This layer also covers operational concerns (e.g., redis, postgresql, web server start scripts) and supports cross-platform installation (Ubuntu, Windows exe, Docker).
Out Relationships and Interactions
• Data Flow:
• The frontend (client) interacts with the backend via REST APIs or WebSocket connections.
• The backend is split between the AI service (handling natural language conversations and query generation) and the BI service (handling dashboard assembly, authentication, and query management). They may interact internally (e.g., the AI’s selected persistent queries being passed to the BI layer for visualization).
• Both backend layers connect to various external databases and file-based data sources.
• Inter-component communication:
• The AI agents (in “ai/agents”) interact with language model APIs via the “oai” adapters and work in tandem with the backend analytics (e.g., “ai/backend/aidb/analysis”).
• CLI tools in the “bi/cli” directory indicate potential command-line interactions for administration.
Architectural Patterns and Design Principles
• Modularity: The project clearly separates concerns – separate directories for AI logic, BI business services, and a fully formed client. Each module is further broken down into specialized sub-modules (such as different types of agents, specific database adapters, and detailed UI components).
• Microservice/Multi-layer Architecture: Although it’s a monorepo, the distinct “ai” and “bi” modules suggest a layered approach where conversational NLP and data analytics are encapsulated. The use of Docker and docker-compose further hints at containerized deployment, which supports scalability.
• Extensibility and Pluggability: With numerous adapters (e.g., for different LLM providers under “ai/agents/oai”) and data source modules, the architecture is designed to be extended for new databases, languages, or visualization types.
Relevant Technologies and Frameworks
• Python: For backend services and AI agents.
• JavaScript/React: For the frontend, including a wide array of UI components and visualization tools.
• Docker & docker-compose: For containerized deployment of the complete stack.
• Database connectors and ORM/analysis libraries: As seen in multiple “analysis” and “report” modules.
• Redis & PostgreSQL: Indicated in the Ubuntu installation instructions for caching and database management.
Being Detailed
• Similarly, represent the CLI, authentication, and tasks in the “bi” module as separate blocks if desired.
• If space permits, incorporate deployment and integration scripts (e.g., Docker, Ubuntu installation scripts) to reflect the operational aspect of the system.