How digai-co/datagpt works
Project type and purpose
• DataGPT is a full-stack AI Agent web application.
• Purpose: Accept natural-language queries via a browser UI, translate them into SQL via LLM, fetch data from a relational database, generate charts, and display results.
File structure
• Top-level:
• server/app.py – entrypoint for the web server and HTTP routes.
• datagpt/ – core Python package implementing the AI Agent.
• config/config.yaml – runtime configuration for OpenAI, database, and memory.
• examples/ – sample database schemas, scripts to spin up MySQL/Postgres.
• Dockerfile, Makefile – containerization and build automation.
• requirements.txt – Python dependencies.
• datagpt/ subpackages:
• action/ – “skills” the agent can perform: text2sql, fetch_data, data2chart.
• tool/db/ – database adapters (MySQL, Postgres).
• tool/llm_openai.py – wrapper around the OpenAI API.
• memory/faiss.py – vector store for schema and session memory.
• util.py, config.py, log.py – utilities, configuration loader, logging.
• datagpt.py – orchestrator gluing LLM, tools, actions into an agent loop.
README for additional insights
• Confirms architecture: web front end → backend application → LLM calls → DB queries → chart rendering.
• Mentions design principles (simplicity, AI Agent pattern) and key features (NL queries, visualization, web UI).
a. Main components
1. User Interface (browser) – web page for input and chart display.
2. Web Server (server/app.py + Flask/FastAPI) – HTTP endpoints.
3. Agent Orchestrator (datagpt/datagpt.py) – core loop interpreting user intent.
4. LLM Service (OpenAI API) – hosted AI model for text2sql and planning.
5. Database (PostgreSQL/MySQL) – stores domain data.
6. Memory Store (Faiss vector DB) – caches schema, previous interactions.
7. Chart Generator (matplotlib/plotly inside action/data2chart.py).
8. Configuration & Logging – config/config.yaml and datagpt/log.py.
9. Examples & DB Setup Scripts – for quickstart environments (examples/db).
b. Relationships and data flow
• User → HTTP Request → Web Server
• Server loads configuration → passes query to Agent
• Agent invokes text2sql → calls OpenAI API → generates SQL
• Agent calls fetch_data → DB adapter → executes SQL → returns rows
• Agent calls data2chart → generates chart image or JSON
• Server returns chart to UI
• Agent writes/reads schema & session embeddings in Faiss
c. Architectural patterns and principles
• AI Agent pattern: decoupled “actions” invoked dynamically.
• Modular design: clear separation between LLM interface, DB tools, and actions.
• Config-driven: supports multiple DBs and LLM endpoints.
• Simple monolith: single codebase, no microservices but clear logical modules.
d. Key technologies
• Python, Flask (or similar) for server
• OpenAI API for LLM calls
• SQLAlchemy or raw connectors in datagpt/tool/db for DB
• Faiss for vector memory
• Matplotlib/Plotly for charts
• Docker for containerization
Detail
• Every logical component: config loader, logging, memory, each action module.
• Sequence: user input → planning → execution → response.
• Avoid over-splitting: group related files under one component (e.g., all datagpt/tool/db files under “DB Adapter”).