How abhinavyadav11/ai-dataanalyst-agent works
Project Type and Purpose
• This is a monolithic, full-stack Streamlit application (“AI File Analyst”) written in Python.
• Purpose: Allow end users to upload various file types (.csv, .pdf, .docx, images, etc.), extract text, and perform LLM-based operations (summaries, Q&A, keyword/entities) via external APIs (OpenAI, Together AI).
File Structure
• Top-level files:
• .devcontainer/devcontainer.json – VS Code dev container configuration
• app1.py – main Streamlit application entrypoint
• requirements.txt – Python dependencies
• README.md, LICENSE
• No separate “frontend” vs “backend” directories – it’s a single Streamlit app.
• Dependencies include: PyMuPDF/pdfplumber (PDF parsing), python-docx, pandas, Pillow (image handling), plus LLM clients.
Key Insights from README
• Interactive UI built with Streamlit
• Supports file‐upload widget → local extraction pipeline → LLM calls → display results
• Secret management via .streamlit/secrets.toml for API keys
• Deployment target: Streamlit Cloud (or local devcontainer)
Main Components
a. Streamlit UI Layer (app1.py):
• File Upload Widget
• User controls (buttons for summary, Q&A)
• Results display panels
b. File Parsing Module(s):
• CSV/Excel parser (pandas)
• PDF extractor (PyMuPDF/pdfplumber)
• DOCX parser (python-docx)
• OCR pipeline (Pillow + pytesseract or similar)
c. Text Processing & Orchestration:
• Cleans/extracts consolidated text
• Routes request to appropriate LLM operation (summarize, keywords, Q&A)
d. External LLM Service Clients:
• OpenAI API client
• Together AI API client
• Handled via Python HTTP calls using API keys from Secrets Manager
e. Secrets/Config Management:
• .streamlit/secrets.toml or environment variables
• Accessed at runtime by Streamlit’s st.secrets
f. Dev/Deployment Infra:
• Devcontainer (.devcontainer/) for local environment
• Streamlit Cloud or containerized hosting
Relationships and Data Flows
• User → (1) Streamlit UI → (2) File Parsing Module → (3) Text Aggregator
• (3) Text → (4) LLM Client → External API (OpenAI/Together AI)
• External API → (5) Response → (6) UI Display
• Secrets Manager feeds API keys into LLM Client
• Devcontainer encapsulates entire stack for local dev
Architectural Patterns & Design Principles
• Single-page, event-driven UI (Streamlit callbacks)
• Modular pipeline for file-type handling (strategy pattern for parsers)
• Separation of concerns: UI vs parsing vs external API integration
• Configuration-as-code (.devcontainer, secrets.toml)
• Stateless: no persistent database; all state lives in session memory