# kernelshreyak/agentic-data-query-engine architecture

> Architecture diagram of the GitHub repository kernelshreyak/agentic-data-query-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kernelshreyak/agentic-data-query-engine
- Repository: https://github.com/kernelshreyak/agentic-data-query-engine

## Overview

##### Project Type and Purpose
- ADQE is a specialized backend query engine designed to work with diverse, locally readable data sources (including files like CSV, JSON, etc., and SQL databases).
- It uses “agents” (such as analysis and querying agents) to choose and execute the best querying mechanism automatically, including capabilities like RAG (Retrieval-Augmented Generation) and code/tool execution.
- The README also mentions that it leverages OpenAI GPT-4, which indicates a reliance on LLM-powered logic for query formulation.

##### Analyzing the File Structure
- Top-Level Files and Directories:
- README.md and images: Provide project documentation and architectural visualization.
- adqe_base/: This is the core package containing:
- adqe_agents/: Houses files related to agent logic (analysis_agents.py and querying_agents.py).
- Other modules like agent_tasks.py, agent_teams.py, agent_tools.py, background_task.py, and global_prompts.py that structure the agent workflow and supporting functions.
- adqe_server.py: Likely the entry point for the server (API) part of the application.
- data/ and data_models.py: Represent data sources and possibly the schema definitions.
- migration.py, requirements.txt, and start.sh: Present setup, deployment, and migration support.
- test/ directory: Contains unit and integration tests ensuring system functionality.

##### Key Components to Highlight
a. Data Source Layer:
- Data files (e.g., population_data.json, rag_input_test.json, scoobydoo.csv, titanic.json) loaded from the “data” directory.
- SQL databases (implied via README) with potential connectors managed by agents.

b. Agentic Query Engine (Core):
- adqe_agents: Different agents that determine the best querying strategy. Show separate blocks for “Analysis Agents” and “Querying Agents”.
- agent_tasks, agent_teams, agent_tools: Additional modules indicating orchestration, team-management, and tool integrations for executing queries.
- global_prompts: Likely used to provide standardized inputs or instructions for agent behavior.

c. Execution & Backend:
- adqe_server.py: Acts as the frontend API/server component that routes requests to the query engine.
- background_task.py: Represents asynchronous tasks or background execution with possibly a job queue (using RQ as indicated by the README instructions to start the RQ worker).
- migration.py: Handles database or schema migrations.

d. Supporting Infrastructure:
- Test modules: Ensure reliability and correctness of different components.
- External Dependencies: OpenAI’s GPT-4 engine is utilized, so indicate this external service integration.
- Job Queue: The RQ worker (with scheduler) manages background query executions.

##### Architectural Relationships and Interactions
- How the adqe_server.py receives incoming query requests which are routed to the agentic query engine.
- The agents (analysis and querying) determine the appropriate method to process the data – whether running direct file queries or using external SQL connectors.
- Flow where background_task might offload heavy processing to separate workers via the RQ queue.
- That the agents may use global prompts and agent_tools to interface with external APIs (like GPT-4) or perform tool executions.

##### Important Architectural Patterns and Principles
- Modular design: Separation between agent logic (adqe_agents) and orchestration (tasks, teams, tools).
- Asynchronous/background processing: With explicit support for RQ-based task scheduling and execution.
- Agnostic data sourcing: Clearly show that the system is designed to work with multiple data types and sources.
- Extensibility: The project structure implies that adding new agents or data connectors is possible.

## Diagram (Mermaid)

```mermaid
flowchart TD

    subgraph "Client/Request Interface"
        API["adqe_server.py"]:::api
    end

    subgraph "Agentic Query Engine"
        AA["Analysis Agents"]:::agent
        QA["Querying Agents"]:::agent
        TASK["Agent Tasks"]:::agent
        TEAMS["Agent Teams"]:::agent
        TOOLS["Agent Tools"]:::agent
        GP["Global Prompts"]:::agent
    end

    subgraph "Data Sources"
        DS["Data Files"]:::data
        DM["Data Models"]:::data
    end

    subgraph "Background Processing"
        BG["Background Task"]:::bg
        MIG["Migration"]:::bg
    end

    subgraph "Supporting Infrastructure"
        TEST["Testing"]:::support
        CONFIG["External Configuration"]:::support
        DOC["Documentation"]:::support
        DEPS["Dependencies & Setup"]:::support
    end

    subgraph "External AI Service"
        GPT["GPT-4"]:::external
    end

    API -->|"routes_query"| AA
    API -->|"routes_query"| QA
    AA -->|"calls"| GP
    QA -->|"calls"| GP
    AA -->|"orchestrates"| TASK
    QA -->|"orchestrates"| TASK
    TASK -->|"manages_team"| TEAMS
    TASK -->|"invokes_tools"| TOOLS
    AA -->|"queries"| DS
    QA -->|"queries"| DS
    DS -->|"provides_schema"| DM
    API -->|"triggers"| BG
    BG -->|"executes"| MIG
    AA -->|"integrates_LLM"| GPT
    QA -->|"integrates_LLM"| GPT

    click API "https://github.com/kernelshreyak/agentic-data-query-engine/blob/main/adqe_server.py"
    click AA "https://github.com/kernelshreyak/agentic-data-query-engine/blob/main/adqe_base/adqe_agents/analysis_agents.py"
    click QA "https://github.com/kernelshreyak/agentic-data-query-engine/blob/main/adqe_base/adqe_agents/querying_agents.py"
    click TASK "https://github.com/kernelshreyak/agentic-data-query-engine/blob/main/adqe_base/agent_tasks.py"
    click TEAMS "https://github.com/kernelshreyak/agentic-data-query-engine/blob/main/adqe_base/agent_teams.py"
    click TOOLS "https://github.com/kernelshreyak/agentic-data-query-engine/blob/main/adqe_base/agent_tools.py"
    click GP "https://github.com/kernelshreyak/agentic-data-query-engine/blob/main/adqe_base/global_prompts.py"
    click BG "https://github.com/kernelshreyak/agentic-data-query-engine/blob/main/adqe_base/background_task.py"
    click MIG "https://github.com/kernelshreyak/agentic-data-query-engine/blob/main/migration.py"
    click DS "https://github.com/kernelshreyak/agentic-data-query-engine/tree/main/data/"
    click DM "https://github.com/kernelshreyak/agentic-data-query-engine/blob/main/data_models.py"
    click TEST "https://github.com/kernelshreyak/agentic-data-query-engine/tree/main/test/"
    click CONFIG "https://github.com/kernelshreyak/agentic-data-query-engine/blob/main/agents.yaml"
    click DOC "https://github.com/kernelshreyak/agentic-data-query-engine/blob/main/README.md"
    click DEPS "https://github.com/kernelshreyak/agentic-data-query-engine/blob/main/requirements.txt"

    classDef api fill:#f96,stroke:#333,stroke-width:2px;
    classDef agent fill:#bbf,stroke:#333,stroke-width:2px;
    classDef data fill:#cfc,stroke:#333,stroke-width:2px;
    classDef bg fill:#fcf,stroke:#333,stroke-width:2px;
    classDef support fill:#ffc,stroke:#333,stroke-width:2px;
    classDef external fill:#9cf,stroke:#333,stroke-width:2px;
```
