How raisul587/auto_data_ju works
Project Type and Purpose
• This is a full-stack, browser-based data analysis and machine learning platform built with Streamlit.
• Purpose: provide end-to-end data ingestion, cleaning, exploration, visualization, ML training/forecasting, and dashboarding.
File Structure Analysis
• Top-level entry: app.py (Streamlit server entry).
• Configuration: .streamlit/config.toml for theming.
• Pages (under pages/): each page implements one UI module – home (data upload), data cleaning, exploration, visualization, modeling (ML), and dashboard.
• Utilities (utils/): data ingestion & caching (data_utils.py), filtering (filter_utils.py), feature engineering, ML routines (ml_utils.py), plotting helpers.
• Models cache (models/): persisted dataset, future model artifacts.
• Components (components/): custom Streamlit components (if any).
• Requirements, README, CHANGELOG, and filtering guide docs sit at root.
Key Architecture Insights from README
• Multi-source ingestion via Pandas/SQLAlchemy.
• Global filter sidebar affecting all pages (in-memory SQL).
• Caching persisted in models/cached_dataset.pkl.
• Visualization via Plotly/Matplotlib/Seaborn.
• ML with scikit-learn, XGBoost, PyCaret, Prophet.
• Dashboard aggregates saved visualizations and metrics.
Main Components
a. Client Layer
• Browser/UI – HTTP/WebSocket client interacting with Streamlit server.
b. Presentation Layer
• Streamlit Server (app.py) – hosts the UI, routes to pages.
c. Page Modules (logical sub-components)
• Home Page (data upload & DB connector)
• Data Page (cleaning & transformations)
• Exploration Page (EDA & statistics)
• Visualization Page (charts & saving)
• Modeling Page (feature engineering, model training, forecasting)
• Dashboard Page (KPIs & saved charts)
d. Business Logic Layer
• utils/data_utils: ingestion, caching, session persistence
• utils/filter_utils: global filter application
• utils/feature_engineering: scaling, encoding, transformations
• utils/ml_utils: model orchestration, metrics, predictions
• utils/plotting: standardized chart rendering
e. Data Layer
• In-memory DataFrame store
• Persistent cache: models/cached_dataset.pkl
• External databases via SQLAlchemy (PostgreSQL/MySQL/SQLite)
f. Third-Party Services & Libraries
• Pandas/NumPy for data manipulation
• Plotly / Matplotlib / Seaborn for visualization
• scikit-learn, XGBoost, PyCaret, Prophet for ML
• SQLAlchemy for DB connectivity
g. Configuration & Deployment
• Streamlit config (.streamlit/config.toml)
• requirements.txt for dependencies
Component Interactions and Data Flows
• User uploads data → Streamlit Home Page → data_utils loads into DataFrame → cache write/read
• Sidebar filters → filter_utils transforms DataFrame → updated view on all pages
• Data cleaning actions (Data Page) → data_utils & feature_engineering → DataFrame mutation
• Exploration & Visualization pages invoke plotting utilities → render charts → optionally save state to Dashboard store
• Modeling page calls feature_engineering & ml_utils → train models → metrics & forecasts → return metrics & plots
• Dashboard aggregates saved visualizations and model metrics via ml_utils & plotting
• All writes (cache or saved charts) persist to local storage under models/
Architectural Patterns & Principles
• Layered architecture (Presentation → Business Logic → Data)
• Modular page-based design (each UI page is a self-contained component)
• Utility modules encapsulate cross-cutting concerns (filtering, ML, plotting)
• In-memory caching for performance, with persistent fallback
• Separation of concerns: UI, logic, data handling