# nguyentuongbachhy/data-visualization-with-python architecture

> Architecture diagram of the GitHub repository nguyentuongbachhy/data-visualization-with-python, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nguyentuongbachhy/data-visualization-with-python
- Repository: https://github.com/nguyentuongbachhy/data-visualization-with-python

## Overview

##### Main Components
- Data Sources & Storage:
- Raw Data: Represent the “data/raw” folder (containing datasets from Kaggle such as “covid_19_data.csv”, “time_series_covid_19_confirmed.csv”, etc.).
- Processed Data: Represent the “data/processed” folder, including daily data CSVs and data quality issues.
- Data Processing & Analysis:
- Data Collection Notebook: “src/data_collection.ipynb” handles initial data ingestion and exploratory tasks.
- Data Cleaning Notebook: “src/data_cleaning.ipynb” performs data quality checking and cleaning (handle missing, negative, and “Unknown” values).
- Data Processing Notebook: “src/data_processing.ipynb” further transforms data for analysis.
- Modeling / Forecasting:
- Model Notebooks & Models Folder: “src/model.ipynb” along with the “src/models” directory. Here, separate subcomponents exist for forecasting confirmed cases, deaths, and recoveries using LSTM (Keras models) and Prophet (with saved models, params, and forecast CSVs). Also include the ensemble script (“prophet_lstm_ensemble.py”) if applicable.
- Visualization & Reporting:
- Data Visualization Notebook: “src/data_visualization.ipynb” creates various plots—line charts, bar charts, heatmaps, choropleth maps, animated plots, etc.
- Results: “src/results” folder which holds forecast output CSVs.
- Documentation: “docs” folder containing the project report and README.

##### Relationships and Data Flow
- Input Flow: Data enters from external sources (dataset from Kaggle) and is saved into the “data/raw” folder.
- Processing Pipeline: Notebooks in “src” run sequentially for collection → cleaning → processing.
- Modeling Stage: After cleaning and processing, forecasting models are executed (using LSTM and Prophet), with intermediate models and scaling parameters saved under “src/models/Vietnam_Confirmed”, “Vietnam_Deaths”, and “Vietnam_Recovered”.
- Results Flow: Forecast outputs are stored under “src/results” and then used for visualization.
- Visualization & Reporting: Final visual outputs and dashboards are built in the “src/data_visualization.ipynb” and supported by documentation in “docs”.

##### Architectural Patterns and Technologies
- Modularity: The project is broken into distinct stages (data ingestion, cleaning, modeling, visualization), which is a modular design.
- Notebook-Based Workflow: The use of Jupyter Notebooks for interactive analysis and reporting.
- Model Management: Separate folder structure for different forecasting models, each with its configuration (params.json), trained models (keras, Prophet), and scaling information.
- Technologies: Emphasize Python 3.x, Pandas/NumPy for data manipulation, Matplotlib/Seaborn (and optionally Plotly) for visualization, Keras for LSTM modeling, and Prophet for forecasting.

##### Final Elements
- Clear Labels: Ensure every node (Data Source, Notebook, Model, Result, etc.) has a descriptive title.
- Annotations might include brief notes such as “Data Cleaning: Handling negative values and missing entries using rolling average” or “Forecasting: LSTM and Prophet used for time series analysis.”

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Sources
    subgraph "Data Sources"
        data_raw["Raw Data"]:::data
        data_processed["Processed Data"]:::data
    end

    %% Data Pipeline
    subgraph "Data Pipeline"
        notebook_collection["Data Collection Notebook"]:::notebook
        notebook_cleaning["Data Cleaning Notebook"]:::notebook
        notebook_processing["Data Processing Notebook"]:::notebook
    end

    %% Modeling
    subgraph "Modeling"
        forecasting_notebook["Forecasting Models Notebook"]:::model
        forecasting_artifacts["Forecasting Models & Artifacts"]:::model
        confirmed["Vietnam Confirmed Forecasting"]:::model
        deaths["Vietnam Deaths Forecasting"]:::model
        recovered["Vietnam Recovered Forecasting"]:::model
        ensemble["Ensemble Forecasting Script"]:::model
    end

    %% Visualization & Reporting
    subgraph "Visualization & Reporting"
        visualization_notebook["Data Visualization Module"]:::vis
        results["Visualization Results"]:::vis
        docs["Documentation & Reporting"]:::vis
    end

    %% Data Flow Connections
    data_raw -->|"feeds"| notebook_collection
    notebook_collection -->|"triggers"| notebook_cleaning
    notebook_cleaning -->|"cleans-data-to"| data_processed
    notebook_cleaning -->|"prepares"| notebook_processing
    notebook_processing -->|"transforms"| forecasting_notebook
    forecasting_notebook -->|"executes"| forecasting_artifacts
    forecasting_artifacts -->|"contains"| confirmed
    forecasting_artifacts -->|"contains"| deaths
    forecasting_artifacts -->|"contains"| recovered
    forecasting_notebook -->|"ensemble"| ensemble
    confirmed -->|"outputs-for"| ensemble
    deaths -->|"outputs-for"| ensemble
    recovered -->|"outputs-for"| ensemble
    ensemble -->|"integrates"| visualization_notebook
    forecasting_notebook -->|"prepares-outputs-for"| visualization_notebook
    visualization_notebook -->|"generates"| results
    visualization_notebook -->|"supported-by"| docs

    %% Styles
    classDef data fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef notebook fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef model fill:#C39BD3,stroke:#8E44AD,stroke-width:2px;
    classDef vis fill:#82E0AA,stroke:#2E8B57,stroke-width:2px;

    %% Click Events
    click data_raw "https://github.com/nguyentuongbachhy/data-visualization-with-python/tree/main/data/raw"
    click data_processed "https://github.com/nguyentuongbachhy/data-visualization-with-python/tree/main/data/processed"
    click notebook_collection "https://github.com/nguyentuongbachhy/data-visualization-with-python/blob/main/src/data_collection.ipynb"
    click notebook_cleaning "https://github.com/nguyentuongbachhy/data-visualization-with-python/blob/main/src/data_cleaning.ipynb"
    click notebook_processing "https://github.com/nguyentuongbachhy/data-visualization-with-python/blob/main/src/data_processing.ipynb"
    click forecasting_notebook "https://github.com/nguyentuongbachhy/data-visualization-with-python/blob/main/src/model.ipynb"
    click forecasting_artifacts "https://github.com/nguyentuongbachhy/data-visualization-with-python/tree/main/src/models"
    click confirmed "https://github.com/nguyentuongbachhy/data-visualization-with-python/tree/main/src/models/Vietnam_Confirmed"
    click deaths "https://github.com/nguyentuongbachhy/data-visualization-with-python/tree/main/src/models/Vietnam_Deaths"
    click recovered "https://github.com/nguyentuongbachhy/data-visualization-with-python/tree/main/src/models/Vietnam_Recovered"
    click ensemble "https://github.com/nguyentuongbachhy/data-visualization-with-python/blob/main/src/models/prophet_lstm_ensemble.py"
    click visualization_notebook "https://github.com/nguyentuongbachhy/data-visualization-with-python/blob/main/src/data_visualization.ipynb"
    click results "https://github.com/nguyentuongbachhy/data-visualization-with-python/tree/main/src/results"
    click docs "https://github.com/nguyentuongbachhy/data-visualization-with-python/tree/main/docs"
```
