# sherlock-221bbs/conversational_agent_on_structured_data architecture

> Architecture diagram of the GitHub repository sherlock-221bbs/conversational_agent_on_structured_data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sherlock-221bbs/conversational_agent_on_structured_data
- Repository: https://github.com/sherlock-221bbs/conversational_agent_on_structured_data

## Overview

##### Project Type and Purpose
- This is essentially a full-stack, Python-based web application with an interactive Streamlit UI on the front end, a stateful LLM-driven agent in the back end, and a file-based data store.
- Its purpose is to let users upload .csv/.xlsx/.pdf files, query them with natural language, execute Pandas code under the hood, and display results or charts in real time.

##### Main Components
a. Frontend (Streamlit UI)
- app.py: Handles file uploads, chat interface, and displays text/plot outputs.
b. Agent Core
- src/agent.py: Orchestrates the Retrieval→Augmentation→Generation pipeline.
- Tools:
- list_available_tables (metadata retrieval)
- query_table (generates & executes Pandas code)
- generate_visualization (executes plotting code)
c. Data Ingestion Layer
- src/data_loader.py: Contains CSV/Excel loaders (Pandas) and the custom PDF parser for the hydrostatic_table.pdf.
d. Data Storage
- data/… folder: Raw files (.csv, .xlsx, .pdf) and sample files.
- In-memory DataStore: holds DataFrame objects and table summaries.
e. Code Execution Environment
- Python interpreter (exec()): runs LLM-generated code snippets against the DataFrames.
f. External Service
- OpenAI API: LLM calls for prompt completion and code generation.

##### Relationships & Data Flow
1. User interacts with Streamlit UI (upload file, type query).
2. app.py passes file to data_loader; DataLoader returns DataFrame(s) to in-memory DataStore.
3. User query → Agent core in src/agent.py.
4. Agent calls list_available_tables → DataStore → returns table metadata.
5. Agent decides which tool to invoke (query_table or generate_visualization).
6. Agent generates Python code (via LLM API) and passes it to the exec() environment.
7. Executed code reads/writes DataFrame, produces results or chart object.
8. Agent collects the result, wraps it in natural language, returns to app.py.
9. Streamlit UI displays text answer or renders chart.

##### Architectural Patterns & Design Principles
- Agentic RAG: Splits “retrieval” (table metadata) and “augmentation” (executable code) from “generation” (LLM response).
- Tool‐based agent: Encapsulates discrete capabilities (listing, querying, plotting) as separate tools.
- Layered design:
- Presentation (Streamlit)
- Orchestration (AgentExecutor)
- Domain logic (DataLoader, DataStore)
- Execution (Python exec)
- Single‐process but logically “microservice‐like” via tool calls.

1. User & UI
2. Agent Core
3. Data & Code Execution
4. External Services
- Streamlit UI (app.py)
- Agent Executor (src/agent.py)
- Tools (list_available_tables, query_table, generate_visualization)
- DataLoader (src/data_loader.py)
- In-memory DataStore
- Python exec environment
- OpenAI API
- File System (data folder)
- UI → Agent (queries)
- UI → DataLoader (file ingestion)
- Agent → list_available_tables → DataStore → back to Agent
- Agent → query_table → exec() → DataStore → back to Agent
- Agent → generate_visualization → exec() → returns plot → back to Agent
- Agent → UI (final response)
- Agent ↔ OpenAI API (prompt → completion → code/text)
- Green for core logic (Agent Executor, Tools)
- Orange for data components (DataLoader, DataStore, File System)
- Purple for external (OpenAI API)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User & UI Layer
    subgraph "User & UI"
        UI["Streamlit UI (app.py)"]:::frontend
    end

    %% Agent Core Layer
    subgraph "Agent Core"
        AgentExecutor["Agent Executor\n(src/agent.py)"]:::core
        ListTables["Tool: list_available_tables"]:::core
        QueryTable["Tool: query_table"]:::core
        GenerateViz["Tool: generate_visualization"]:::core
        InMemoryStore["In-Memory DataStore"]:::data
        CodeExec["Code Execution Environment\n(exec())"]:::data
    end

    %% Data & Code Execution Layer
    subgraph "Data & Code Execution"
        DataLoader["Data Ingestion Layer\n(src/data_loader.py)"]:::data
        FileSystem["File System\n(data/)"]:::data
    end

    %% External Services Layer
    subgraph "External Services"
        OpenAI["OpenAI API"]:::external
    end

    %% Connections
    UI -->|"upload file"| DataLoader
    DataLoader -->|"DataFrame(s)"| InMemoryStore
    UI -->|"user query"| AgentExecutor

    AgentExecutor -->|"list tables"| ListTables
    ListTables -->|"metadata"| InMemoryStore
    InMemoryStore -->|"table metadata"| AgentExecutor

    AgentExecutor -->|"query_table"| QueryTable
    QueryTable -->|"code snippet"| CodeExec
    CodeExec -->|"exec against DataStore"| InMemoryStore
    InMemoryStore -->|"query result"| AgentExecutor

    AgentExecutor -->|"generate_visualization"| GenerateViz
    GenerateViz -->|"plot code"| CodeExec
    CodeExec -->|"chart object"| AgentExecutor

    AgentExecutor -->|"final response"| UI

    AgentExecutor -->|"prompt & code gen"| OpenAI
    OpenAI -->|"LLM completion"| AgentExecutor

    %% Click Events
    click UI "https://github.com/sherlock-221bbs/conversational_agent_on_structured_data/blob/main/app.py"
    click AgentExecutor "https://github.com/sherlock-221bbs/conversational_agent_on_structured_data/blob/main/src/agent.py"
    click ListTables "https://github.com/sherlock-221bbs/conversational_agent_on_structured_data/blob/main/src/agent.py"
    click QueryTable "https://github.com/sherlock-221bbs/conversational_agent_on_structured_data/blob/main/src/agent.py"
    click GenerateViz "https://github.com/sherlock-221bbs/conversational_agent_on_structured_data/blob/main/src/agent.py"
    click DataLoader "https://github.com/sherlock-221bbs/conversational_agent_on_structured_data/blob/main/src/data_loader.py"
    click FileSystem "https://github.com/sherlock-221bbs/conversational_agent_on_structured_data/tree/main/data/"
    click InMemoryStore "https://github.com/sherlock-221bbs/conversational_agent_on_structured_data/blob/main/src/agent.py"
    click CodeExec "https://github.com/sherlock-221bbs/conversational_agent_on_structured_data/blob/main/src/agent.py"
    click OpenAI "https://github.com/sherlock-221bbs/conversational_agent_on_structured_data/blob/main/src/agent.py"

    %% Styles
    classDef frontend fill:#D0E8FF,stroke:#0366D6,color:#0366D6
    classDef core fill:#E6FFDC,stroke:#28A745,color:#28A745
    classDef data fill:#FFF4D6,stroke:#CC7A00,color:#CC7A00
    classDef external fill:#F0E6FF,stroke:#6F42C1,color:#6F42C1
```
