How nguyentuongbachhy/data-visualization-with-python works
Main Components
• Data Sources & Storage:
• Raw Data: Represent the “data/raw” folder (containing datasets from Kaggle such as “covid_19_data.csv”, “time_series_covid_19_confirmed.csv”, etc.).
• Processed Data: Represent the “data/processed” folder, including daily data CSVs and data quality issues.
• Data Processing & Analysis:
• Data Collection Notebook: “src/data_collection.ipynb” handles initial data ingestion and exploratory tasks.
• Data Cleaning Notebook: “src/data_cleaning.ipynb” performs data quality checking and cleaning (handle missing, negative, and “Unknown” values).
• Data Processing Notebook: “src/data_processing.ipynb” further transforms data for analysis.
• Modeling / Forecasting:
• Model Notebooks & Models Folder: “src/model.ipynb” along with the “src/models” directory. Here, separate subcomponents exist for forecasting confirmed cases, deaths, and recoveries using LSTM (Keras models) and Prophet (with saved models, params, and forecast CSVs). Also include the ensemble script (“prophet_lstm_ensemble.py”) if applicable.
• Visualization & Reporting:
• Data Visualization Notebook: “src/data_visualization.ipynb” creates various plots—line charts, bar charts, heatmaps, choropleth maps, animated plots, etc.
• Results: “src/results” folder which holds forecast output CSVs.
• Documentation: “docs” folder containing the project report and README.
Relationships and Data Flow
• Input Flow: Data enters from external sources (dataset from Kaggle) and is saved into the “data/raw” folder.
• Processing Pipeline: Notebooks in “src” run sequentially for collection → cleaning → processing.
• Modeling Stage: After cleaning and processing, forecasting models are executed (using LSTM and Prophet), with intermediate models and scaling parameters saved under “src/models/Vietnam_Confirmed”, “Vietnam_Deaths”, and “Vietnam_Recovered”.
• Results Flow: Forecast outputs are stored under “src/results” and then used for visualization.
• Visualization & Reporting: Final visual outputs and dashboards are built in the “src/data_visualization.ipynb” and supported by documentation in “docs”.
Architectural Patterns and Technologies
• Modularity: The project is broken into distinct stages (data ingestion, cleaning, modeling, visualization), which is a modular design.
• Notebook-Based Workflow: The use of Jupyter Notebooks for interactive analysis and reporting.
• Model Management: Separate folder structure for different forecasting models, each with its configuration (params.json), trained models (keras, Prophet), and scaling information.
• Technologies: Emphasize Python 3.x, Pandas/NumPy for data manipulation, Matplotlib/Seaborn (and optionally Plotly) for visualization, Keras for LSTM modeling, and Prophet for forecasting.
Final Elements
• Clear Labels: Ensure every node (Data Source, Notebook, Model, Result, etc.) has a descriptive title.
• Annotations might include brief notes such as “Data Cleaning: Handling negative values and missing entries using rolling average” or “Forecasting: LSTM and Prophet used for time series analysis.”