# raisul587/auto_data_ju architecture

> Architecture diagram of the GitHub repository raisul587/auto_data_ju, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/raisul587/auto_data_ju
- Repository: https://github.com/raisul587/auto_data_ju

## Overview

##### Project Type and Purpose
- This is a full-stack, browser-based data analysis and machine learning platform built with Streamlit.
- Purpose: provide end-to-end data ingestion, cleaning, exploration, visualization, ML training/forecasting, and dashboarding.

##### File Structure Analysis
- Top-level entry: `app.py` (Streamlit server entry).
- Configuration: `.streamlit/config.toml` for theming.
- Pages (under `pages/`): each page implements one UI module – home (data upload), data cleaning, exploration, visualization, modeling (ML), and dashboard.
- Utilities (`utils/`): data ingestion & caching (`data_utils.py`), filtering (`filter_utils.py`), feature engineering, ML routines (`ml_utils.py`), plotting helpers.
- Models cache (`models/`): persisted dataset, future model artifacts.
- Components (`components/`): custom Streamlit components (if any).
- Requirements, README, CHANGELOG, and filtering guide docs sit at root.

##### Key Architecture Insights from README
- Multi-source ingestion via Pandas/SQLAlchemy.
- Global filter sidebar affecting all pages (in-memory SQL).
- Caching persisted in `models/cached_dataset.pkl`.
- Visualization via Plotly/Matplotlib/Seaborn.
- ML with scikit-learn, XGBoost, PyCaret, Prophet.
- Dashboard aggregates saved visualizations and metrics.

##### Main Components
a. Client Layer
- Browser/UI – HTTP/WebSocket client interacting with Streamlit server.
b. Presentation Layer
- Streamlit Server (app.py) – hosts the UI, routes to pages.
c. Page Modules (logical sub-components)
- Home Page (data upload & DB connector)
- Data Page (cleaning & transformations)
- Exploration Page (EDA & statistics)
- Visualization Page (charts & saving)
- Modeling Page (feature engineering, model training, forecasting)
- Dashboard Page (KPIs & saved charts)
d. Business Logic Layer
- utils/data_utils: ingestion, caching, session persistence
- utils/filter_utils: global filter application
- utils/feature_engineering: scaling, encoding, transformations
- utils/ml_utils: model orchestration, metrics, predictions
- utils/plotting: standardized chart rendering
e. Data Layer
- In-memory DataFrame store
- Persistent cache: `models/cached_dataset.pkl`
- External databases via SQLAlchemy (PostgreSQL/MySQL/SQLite)
f. Third-Party Services & Libraries
- Pandas/NumPy for data manipulation
- Plotly / Matplotlib / Seaborn for visualization
- scikit-learn, XGBoost, PyCaret, Prophet for ML
- SQLAlchemy for DB connectivity
g. Configuration & Deployment
- Streamlit config (.streamlit/config.toml)
- requirements.txt for dependencies

##### Component Interactions and Data Flows
- User uploads data → Streamlit Home Page → data_utils loads into DataFrame → cache write/read
- Sidebar filters → filter_utils transforms DataFrame → updated view on all pages
- Data cleaning actions (Data Page) → data_utils & feature_engineering → DataFrame mutation
- Exploration & Visualization pages invoke plotting utilities → render charts → optionally save state to Dashboard store
- Modeling page calls feature_engineering & ml_utils → train models → metrics & forecasts → return metrics & plots
- Dashboard aggregates saved visualizations and model metrics via ml_utils & plotting
- All writes (cache or saved charts) persist to local storage under `models/`

##### Architectural Patterns & Principles
- Layered architecture (Presentation → Business Logic → Data)
- Modular page-based design (each UI page is a self-contained component)
- Utility modules encapsulate cross-cutting concerns (filtering, ML, plotting)
- In-memory caching for performance, with persistent fallback
- Separation of concerns: UI, logic, data handling

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client Layer
    Browser["Browser UI"]:::ui

    %% Presentation Layer
    Streamlit["Streamlit Server"]:::server

    Browser -->|"HTTP"| Streamlit

    %% Page Modules
    subgraph "Streamlit Server"
        Home["Home Page"]:::page
        DataPage["Data Page"]:::page
        Exploration["Exploration Page"]:::page
        Visualization["Visualization Page"]:::page
        Modeling["Modeling Page"]:::page
        Dashboard["Dashboard Page"]:::page
    end

    Streamlit --> Home
    Streamlit --> DataPage
    Streamlit --> Exploration
    Streamlit --> Visualization
    Streamlit --> Modeling
    Streamlit --> Dashboard

    %% Business Logic Layer
    subgraph "Business Logic Utils"
        DataUtils["data_utils"]:::logic
        FilterUtils["filter_utils"]:::logic
        FeatureEng["feature_engineering"]:::logic
        MLUtils["ml_utils"]:::logic
        Plotting["plotting"]:::logic
    end

    Home -->|"calls"| DataUtils
    Home -->|"calls"| FilterUtils

    DataPage -->|"calls"| DataUtils
    DataPage -->|"calls"| FeatureEng
    DataPage -->|"calls"| FilterUtils

    Exploration -->|"calls"| DataUtils
    Exploration -->|"calls"| Plotting
    Exploration -->|"calls"| FilterUtils

    Visualization -->|"calls"| Plotting
    Visualization -->|"calls"| DataUtils

    Modeling -->|"calls"| FeatureEng
    Modeling -->|"calls"| MLUtils
    Modeling -->|"calls"| DataUtils

    Dashboard -->|"calls"| MLUtils
    Dashboard -->|"calls"| Plotting
    Dashboard -->|"calls"| DataUtils

    %% Data Layer
    subgraph "Data Layer"
        InMemory["In-memory DataFrame"]:::data
        Cache["persistent cache\n(cached_dataset.pkl)"]:::data
        ModelsDir["models/"]:::data
        ExtDB["External DB\n(SQLAlchemy)"]:::external
    end

    DataUtils --> InMemory
    DataUtils --> Cache
    DataUtils --> ModelsDir
    DataUtils --> ExtDB

    FilterUtils --> InMemory
    FeatureEng --> InMemory
    MLUtils --> InMemory
    MLUtils --> Cache
    Plotting --> InMemory

    %% Deployment & Config
    subgraph "Deployment & Config"
        Config[".streamlit/config.toml"]:::deploy
        Requirements["requirements.txt"]:::deploy
    end

    Streamlit --> Config
    Streamlit --> Requirements

    %% Click Events
    click Streamlit "https://github.com/raisul587/auto_data_ju/blob/main/app.py"
    click Home "https://github.com/raisul587/auto_data_ju/blob/main/pages/home.py"
    click DataPage "https://github.com/raisul587/auto_data_ju/blob/main/pages/data.py"
    click Exploration "https://github.com/raisul587/auto_data_ju/blob/main/pages/exploration.py"
    click Visualization "https://github.com/raisul587/auto_data_ju/blob/main/pages/visualization.py"
    click Modeling "https://github.com/raisul587/auto_data_ju/blob/main/pages/modeling.py"
    click Dashboard "https://github.com/raisul587/auto_data_ju/blob/main/pages/dashboard.py"
    click DataUtils "https://github.com/raisul587/auto_data_ju/blob/main/utils/data_utils.py"
    click FilterUtils "https://github.com/raisul587/auto_data_ju/blob/main/utils/filter_utils.py"
    click FeatureEng "https://github.com/raisul587/auto_data_ju/blob/main/utils/feature_engineering.py"
    click MLUtils "https://github.com/raisul587/auto_data_ju/blob/main/utils/ml_utils.py"
    click Plotting "https://github.com/raisul587/auto_data_ju/blob/main/utils/plotting.py"
    click Cache "https://github.com/raisul587/auto_data_ju/blob/main/models/cached_dataset.pkl"
    click ModelsDir "https://github.com/raisul587/auto_data_ju/tree/main/models/"
    click Config "https://github.com/raisul587/auto_data_ju/blob/main/.streamlit/config.toml"
    click Requirements "https://github.com/raisul587/auto_data_ju/blob/main/requirements.txt"

    %% Styles
    classDef ui fill:#D0E8FF,stroke:#0366D6
    classDef server fill:#BEE5D8,stroke:#228B22
    classDef page fill:#E0F7FA,stroke:#00838F
    classDef logic fill:#E6F4EA,stroke:#2E8B57
    classDef data fill:#FFE0B2,stroke:#FF8C00
    classDef external fill:#F0F0F0,stroke:#A0A0A0
    classDef deploy fill:#F3E5F5,stroke:#8E24AA
```
