# theshmphony7580/data_agent architecture

> Architecture diagram of the GitHub repository theshmphony7580/data_agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/theshmphony7580/data_agent
- Repository: https://github.com/theshmphony7580/data_agent

## Overview

##### Project Type and Purpose
- This is a full-stack, local AI-powered data analytics assistant.
- Frontend: Streamlit app for user interactions (file upload, NLQ input, report export).
- Backend: FastAPI service exposing endpoints to upload data, profile/clean, generate insights, handle NLQ, build reports.
- ML Engines: Python modules implementing data cleaning, profiling, insights, NLQ stub, report building.
- Data Store: Local CSV/XLSX files in `data/`.

##### File Structure Analysis
- Top-level directories:
- backend/ – FastAPI application
– api/: request handlers (upload.py, profile.py, clean.py, insights.py, nlq.py, report.py)
– ml/: core engines (cleaning.py, profiling.py, insights_engine.py, nlq_engine.py, report_builder.py)
– utils/: shared helpers (file_utils.py, data_utils.py, security.py)
– main.py & config.py: app entry and configuration
- frontend/: Streamlit entrypoint (app.py)
- models/: registry of ML/NLQ models (model_registry.json, trained/)
- data/: persisted uploads & generated reports
- requirements.txt: Python dependencies

##### README Insights
- Backend runs on Uvicorn (FastAPI).
- Frontend runs on Streamlit.
- No external LLM calls yet—NLQ is a placeholder.
- Reports exported as HTML or Excel.
- Environment variables configure backend URL, API keys.

a. Main Components
1. User Interface (Streamlit)
2. API Layer (FastAPI endpoints)
3. ML Engines
- Data Cleaning
- Profiling
- Insights Engine
- NLQ Engine
- Report Builder
4. Utilities
- File I/O
- Data Helpers
- Security
5. Data Store (local filesystem under `data/`)
6. Model Registry (models/model_registry.json & trained/)

b. Component Interactions
1. User → Frontend: uploads dataset, enters NLQ, clicks “Generate Report.”
2. Frontend → Backend API (HTTP requests):
- POST /upload → file stored by file_utils → data/
- GET /profile → calls ml.profiling → returns summary
- POST /clean → ml.cleaning → returns cleaned data
- GET /insights → ml.insights_engine → returns insights
- POST /nlq → ml.nlq_engine → returns code snippet
- GET /report → ml.report_builder → writes HTML/XLSX to data/ → returns download link
3. Backend API → ML Engines (in-process calls)
4. ML Engines → Utilities (file_utils for reading/writing files; data_utils for Pandas operations; security for sanitization)
5. Backend & Frontend share Model Registry via utils or direct read

c. Architectural Patterns & Principles
- Layered Architecture:
- Presentation (Streamlit)
- API / Application (FastAPI)
- Business Logic (ML engines)
- Data Access (file_utils + local FS)
- Separation of Concerns: each ML task in its own module
- Dependency Injection via config.py / environment variables
- Stateless API design; persisted state on disk

d. Key Technologies
- FastAPI (backend HTTP framework)
- Uvicorn (ASGI server)
- Streamlit (interactive frontend)
- Pandas (data profiling/cleaning)
- Python standard libs for IO & security
- Local file system as data store

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        UI["Streamlit UI"]:::frontend
    end

    %% API Layer
    subgraph "API Layer"
        UP["upload.py"]:::api
        PR["profile.py"]:::api
        CL["clean.py"]:::api
        IN["insights.py"]:::api
        NLQ["nlq.py"]:::api
        RP["report.py"]:::api
        MAIN["main.py"]:::api
        CFG["config.py"]:::api
    end

    %% Business Logic (ML Engines)
    subgraph "Business Logic (ML Engines)"
        CLEAN["cleaning.py"]:::ml
        PROF["profiling.py"]:::ml
        INS_ENG["insights_engine.py"]:::ml
        NLQ_ENG["nlq_engine.py"]:::ml
        RPT["report_builder.py"]:::ml
    end

    %% Utilities
    subgraph "Utilities"
        FILEU["file_utils.py"]:::util
        DATAU["data_utils.py"]:::util
        SECU["security.py"]:::util
    end

    %% Storage & Config
    DATA["Data Store (data/)"]:::storage
    MODEL["Model Registry"]:::storage
    REQ["requirements.txt"]:::storage

    %% Interactions
    UI --> UP
    UI --> PR
    UI --> CL
    UI --> IN
    UI --> NLQ
    UI --> RP

    MAIN --> UP
    MAIN --> PR
    MAIN --> CL
    MAIN --> IN
    MAIN --> NLQ
    MAIN --> RP
    CFG --> MAIN

    %% API to ML Engines
    UP --> CLEAN
    PR --> PROF
    CL --> CLEAN
    IN --> INS_ENG
    NLQ --> NLQ_ENG
    RP --> RPT

    %% ML Engines to Utilities & Storage
    CLEAN --> DATAU
    PROF --> DATAU
    INS_ENG --> DATAU
    NLQ_ENG --> MODEL
    RPT --> FILEU
    RPT --> DATA

    %% API Utilities and Data Store
    UP --> FILEU
    UP --> DATAU
    UP --> DATA
    PR --> DATAU
    CL --> DATAU

    %% Class Definitions
    classDef frontend fill:#add8e6,stroke:#000,stroke-width:1px
    classDef api fill:#90ee90,stroke:#000,stroke-width:1px
    classDef ml fill:#ffd580,stroke:#000,stroke-width:1px
    classDef util fill:#d3d3d3,stroke:#000,stroke-width:1px
    classDef storage fill:#cccccc,stroke:#000,stroke-width:1px

    %% Click Events
    click UI "https://github.com/theshmphony7580/data_agent/blob/main/frontend/app.py"
    click UP "https://github.com/theshmphony7580/data_agent/blob/main/backend/api/upload.py"
    click PR "https://github.com/theshmphony7580/data_agent/blob/main/backend/api/profile.py"
    click CL "https://github.com/theshmphony7580/data_agent/blob/main/backend/api/clean.py"
    click IN "https://github.com/theshmphony7580/data_agent/blob/main/backend/api/insights.py"
    click NLQ "https://github.com/theshmphony7580/data_agent/blob/main/backend/api/nlq.py"
    click RP "https://github.com/theshmphony7580/data_agent/blob/main/backend/api/report.py"
    click MAIN "https://github.com/theshmphony7580/data_agent/blob/main/backend/main.py"
    click CFG "https://github.com/theshmphony7580/data_agent/blob/main/backend/config.py"
    click CLEAN "https://github.com/theshmphony7580/data_agent/blob/main/backend/ml/cleaning.py"
    click PROF "https://github.com/theshmphony7580/data_agent/blob/main/backend/ml/profiling.py"
    click INS_ENG "https://github.com/theshmphony7580/data_agent/blob/main/backend/ml/insights_engine.py"
    click NLQ_ENG "https://github.com/theshmphony7580/data_agent/blob/main/backend/ml/nlq_engine.py"
    click RPT "https://github.com/theshmphony7580/data_agent/blob/main/backend/ml/report_builder.py"
    click FILEU "https://github.com/theshmphony7580/data_agent/blob/main/backend/utils/file_utils.py"
    click DATAU "https://github.com/theshmphony7580/data_agent/blob/main/backend/utils/data_utils.py"
    click SECU "https://github.com/theshmphony7580/data_agent/blob/main/backend/utils/security.py"
    click DATA "https://github.com/theshmphony7580/data_agent/tree/main/data/"
    click MODEL "https://github.com/theshmphony7580/data_agent/blob/main/models/model_registry.json"
    click REQ "https://github.com/theshmphony7580/data_agent/blob/main/requirements.txt"
```
