How kernelshreyak/agentic-data-query-engine works
Project Type and Purpose
• ADQE is a specialized backend query engine designed to work with diverse, locally readable data sources (including files like CSV, JSON, etc., and SQL databases).
• It uses “agents” (such as analysis and querying agents) to choose and execute the best querying mechanism automatically, including capabilities like RAG (Retrieval-Augmented Generation) and code/tool execution.
• The README also mentions that it leverages OpenAI GPT-4, which indicates a reliance on LLM-powered logic for query formulation.
Analyzing the File Structure
• Top-Level Files and Directories:
• README.md and images: Provide project documentation and architectural visualization.
• adqe_base/: This is the core package containing:
• adqe_agents/: Houses files related to agent logic (analysis_agents.py and querying_agents.py).
• Other modules like agent_tasks.py, agent_teams.py, agent_tools.py, background_task.py, and global_prompts.py that structure the agent workflow and supporting functions.
• adqe_server.py: Likely the entry point for the server (API) part of the application.
• data/ and data_models.py: Represent data sources and possibly the schema definitions.
• migration.py, requirements.txt, and start.sh: Present setup, deployment, and migration support.
• test/ directory: Contains unit and integration tests ensuring system functionality.
Key Components to Highlight
a. Data Source Layer:
• Data files (e.g., population_data.json, rag_input_test.json, scoobydoo.csv, titanic.json) loaded from the “data” directory.
• SQL databases (implied via README) with potential connectors managed by agents.
b. Agentic Query Engine (Core):
• adqe_agents: Different agents that determine the best querying strategy. Show separate blocks for “Analysis Agents” and “Querying Agents”.
• agent_tasks, agent_teams, agent_tools: Additional modules indicating orchestration, team-management, and tool integrations for executing queries.
• global_prompts: Likely used to provide standardized inputs or instructions for agent behavior.
c. Execution & Backend:
• adqe_server.py: Acts as the frontend API/server component that routes requests to the query engine.
• background_task.py: Represents asynchronous tasks or background execution with possibly a job queue (using RQ as indicated by the README instructions to start the RQ worker).
• migration.py: Handles database or schema migrations.
d. Supporting Infrastructure:
• Test modules: Ensure reliability and correctness of different components.
• External Dependencies: OpenAI’s GPT-4 engine is utilized, so indicate this external service integration.
• Job Queue: The RQ worker (with scheduler) manages background query executions.
Architectural Relationships and Interactions
• How the adqe_server.py receives incoming query requests which are routed to the agentic query engine.
• The agents (analysis and querying) determine the appropriate method to process the data – whether running direct file queries or using external SQL connectors.
• Flow where background_task might offload heavy processing to separate workers via the RQ queue.
• That the agents may use global prompts and agent_tools to interface with external APIs (like GPT-4) or perform tool executions.
Important Architectural Patterns and Principles
• Modular design: Separation between agent logic (adqe_agents) and orchestration (tasks, teams, tools).
• Asynchronous/background processing: With explicit support for RQ-based task scheduling and execution.
• Agnostic data sourcing: Clearly show that the system is designed to work with multiple data types and sources.
• Extensibility: The project structure implies that adding new agents or data connectors is possible.