How theshmphony7580/data_agent works
Project Type and Purpose
• This is a full-stack, local AI-powered data analytics assistant.
• Frontend: Streamlit app for user interactions (file upload, NLQ input, report export).
• Backend: FastAPI service exposing endpoints to upload data, profile/clean, generate insights, handle NLQ, build reports.
• ML Engines: Python modules implementing data cleaning, profiling, insights, NLQ stub, report building.
• Data Store: Local CSV/XLSX files in data/.
File Structure Analysis
• Top-level directories:
• backend/ – FastAPI application
– api/: request handlers (upload.py, profile.py, clean.py, insights.py, nlq.py, report.py)
– ml/: core engines (cleaning.py, profiling.py, insights_engine.py, nlq_engine.py, report_builder.py)
– utils/: shared helpers (file_utils.py, data_utils.py, security.py)
– main.py & config.py: app entry and configuration
• frontend/: Streamlit entrypoint (app.py)
• models/: registry of ML/NLQ models (model_registry.json, trained/)
• data/: persisted uploads & generated reports
• requirements.txt: Python dependencies
README Insights
• Backend runs on Uvicorn (FastAPI).
• Frontend runs on Streamlit.
• No external LLM calls yet—NLQ is a placeholder.
• Reports exported as HTML or Excel.
• Environment variables configure backend URL, API keys.
a. Main Components
1. User Interface (Streamlit)
2. API Layer (FastAPI endpoints)
3. ML Engines
• Data Cleaning
• Profiling
• Insights Engine
• NLQ Engine
• Report Builder
4. Utilities
• File I/O
• Data Helpers
• Security
5. Data Store (local filesystem under data/)
6. Model Registry (models/model_registry.json & trained/)
b. Component Interactions
1. User → Frontend: uploads dataset, enters NLQ, clicks “Generate Report.”
2. Frontend → Backend API (HTTP requests):
• POST /upload → file stored by file_utils → data/
• GET /profile → calls ml.profiling → returns summary
• POST /clean → ml.cleaning → returns cleaned data
• GET /insights → ml.insights_engine → returns insights
• POST /nlq → ml.nlq_engine → returns code snippet
• GET /report → ml.report_builder → writes HTML/XLSX to data/ → returns download link
3. Backend API → ML Engines (in-process calls)
4. ML Engines → Utilities (file_utils for reading/writing files; data_utils for Pandas operations; security for sanitization)
5. Backend & Frontend share Model Registry via utils or direct read
c. Architectural Patterns & Principles
• Layered Architecture:
• Presentation (Streamlit)
• API / Application (FastAPI)
• Business Logic (ML engines)
• Data Access (file_utils + local FS)
• Separation of Concerns: each ML task in its own module
• Dependency Injection via config.py / environment variables
• Stateless API design; persisted state on disk
d. Key Technologies
• FastAPI (backend HTTP framework)
• Uvicorn (ASGI server)
• Streamlit (interactive frontend)
• Pandas (data profiling/cleaning)
• Python standard libs for IO & security
• Local file system as data store