How sherlock-221bbs/conversational_agent_on_structured_data works
Project Type and Purpose
• This is essentially a full-stack, Python-based web application with an interactive Streamlit UI on the front end, a stateful LLM-driven agent in the back end, and a file-based data store.
• Its purpose is to let users upload .csv/.xlsx/.pdf files, query them with natural language, execute Pandas code under the hood, and display results or charts in real time.
Main Components
a. Frontend (Streamlit UI)
• app.py: Handles file uploads, chat interface, and displays text/plot outputs.
b. Agent Core
• src/agent.py: Orchestrates the Retrieval→Augmentation→Generation pipeline.
• Tools:
• list_available_tables (metadata retrieval)
• query_table (generates & executes Pandas code)
• generate_visualization (executes plotting code)
c. Data Ingestion Layer
• src/data_loader.py: Contains CSV/Excel loaders (Pandas) and the custom PDF parser for the hydrostatic_table.pdf.
d. Data Storage
• data/… folder: Raw files (.csv, .xlsx, .pdf) and sample files.
• In-memory DataStore: holds DataFrame objects and table summaries.
e. Code Execution Environment
• Python interpreter (exec()): runs LLM-generated code snippets against the DataFrames.
f. External Service
• OpenAI API: LLM calls for prompt completion and code generation.
Relationships & Data Flow
1. User interacts with Streamlit UI (upload file, type query).
2. app.py passes file to data_loader; DataLoader returns DataFrame(s) to in-memory DataStore.
3. User query → Agent core in src/agent.py.
4. Agent calls list_available_tables → DataStore → returns table metadata.
5. Agent decides which tool to invoke (query_table or generate_visualization).
6. Agent generates Python code (via LLM API) and passes it to the exec() environment.
7. Executed code reads/writes DataFrame, produces results or chart object.
8. Agent collects the result, wraps it in natural language, returns to app.py.
9. Streamlit UI displays text answer or renders chart.
Architectural Patterns & Design Principles
• Agentic RAG: Splits “retrieval” (table metadata) and “augmentation” (executable code) from “generation” (LLM response).
• Tool‐based agent: Encapsulates discrete capabilities (listing, querying, plotting) as separate tools.
• Layered design:
• Presentation (Streamlit)
• Orchestration (AgentExecutor)
• Domain logic (DataLoader, DataStore)
• Execution (Python exec)
• Single‐process but logically “microservice‐like” via tool calls.
1. User & UI
2. Agent Core
3. Data & Code Execution
4. External Services
• Streamlit UI (app.py)
• Agent Executor (src/agent.py)
• Tools (list_available_tables, query_table, generate_visualization)
• DataLoader (src/data_loader.py)
• In-memory DataStore
• Python exec environment
• OpenAI API
• File System (data folder)
• UI → Agent (queries)
• UI → DataLoader (file ingestion)
• Agent → list_available_tables → DataStore → back to Agent
• Agent → query_table → exec() → DataStore → back to Agent
• Agent → generate_visualization → exec() → returns plot → back to Agent
• Agent → UI (final response)
• Agent ↔ OpenAI API (prompt → completion → code/text)
• Green for core logic (Agent Executor, Tools)
• Orange for data components (DataLoader, DataStore, File System)
• Purple for external (OpenAI API)