# gauravssinha/petro_data_forecast architecture

> Architecture diagram of the GitHub repository gauravssinha/petro_data_forecast, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gauravssinha/petro_data_forecast
- Repository: https://github.com/gauravssinha/petro_data_forecast

## Overview

##### Main Components
- Data Collection & Ingestion: Represent the AWS IoT Core (for sensor/device data input), along with the data ingestion scripts located in src/data/ingest_data.py.
- Data Storage: Include Amazon S3 as the central data repository where ingested and processed data are stored.
- ETL Operations: Highlight AWS Glue, which handles ETL tasks to process and transform the raw data.
- Model Development & Training: Use Amazon SageMaker to represent the environment for training machine learning models based on the scripts in src/models/train_model.py and forecasting logic in src/models/forecast.py.
- Visualization & Reporting: Illustrate Amazon QuickSight for dashboard visualization along with the script in src/visualization/create_dashboard.py.
- Monitoring & Logging: Represent Amazon CloudWatch for tracking performance, logs, and alerts.

##### Relationships and Data Flow
- Begin from the data ingestion layer (AWS IoT Core) feeding into Amazon Kinesis for real-time processing.
- Kinesis passes the processed data to both:
- Amazon S3 for storage and later use.
- AWS Glue for ETL tasks to further cleanse and prepare the data.
- The prepared data then flows into Amazon SageMaker where models are trained using the pipelines indicated in src/models.
- The trained model is used by the forecasting script (src/models/forecast.py) to generate predictions.
- Forecast results, along with processed metrics, are visualized in Amazon QuickSight.
- The entire infrastructure setup is initiated via the setup_infrastructure.py script, ensuring configuration and deployment of the needed AWS resources.
- CloudWatch monitors all these components to ensure timely alerts and logs are captured for performance and maintenance.

##### Architectural Patterns and Design Principles
- A modular, service-oriented architecture that clearly separates data ingestion, processing, ML model training, and visualization.
- Separation of concerns (data, models, visualization, utilities) which aligns with best practices seen in the folder structure.
- That AWS services act as managed microservices, which simplifies scaling and integration while maintaining clear boundaries.

##### Relevant Technologies and Frameworks
- AWS IoT Core, S3, Kinesis, Glue, SageMaker, QuickSight, and CloudWatch as core AWS services.
- Mention Python as the core language with libraries listed in requirements.txt, and external dependencies that may be used for data processing, model training, and visualization.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Setup Layer
    A["Infrastructure Setup (src/utils/setup_infrastructure.py)"]:::custom

    %% Data Ingestion & Streaming Layer
    subgraph "Data Ingestion & Streaming"
        B["Data Ingestion (AWS IoT Core / src/data)"]:::aws
        C["Streaming & Real-Time Processing (Amazon Kinesis)"]:::aws
    end

    %% Data Storage & ETL Layer
    subgraph "Data Storage & ETL"
        D["Data Storage (Amazon S3)"]:::aws
        E["ETL Processing (AWS Glue)"]:::aws
    end

    %% Model Development Layer
    subgraph "ML Model Development"
        F["Model Development (Amazon SageMaker)"]:::aws
        G["Training Script (src/models/train_model.py)"]:::custom
        H["Model Definition (src/models/scripts/model_def.py)"]:::custom
    end

    %% Forecasting & Visualization Layer
    subgraph "Forecasting & Visualization"
        I["Forecasting (src/models/forecast.py)"]:::custom
        J["Visualization (Amazon QuickSight)"]:::aws
        K["Dashboard Script (src/visualization/create_dashboard.py)"]:::custom
    end

    %% Monitoring Layer
    subgraph "Monitoring"
        L["Monitoring (Amazon CloudWatch)"]:::aws
    end

    %% Data Flow Connections
    A -->|"configures"| B
    B -->|"feeds"| C
    C -->|"stores"| D
    C -->|"transfers"| E
    D -->|"supplies"| F
    E -->|"prepares"| F
    F -->|"executes"| G
    F -->|"utilizes"| H
    F -->|"triggers"| I
    I -->|"outputs"| J
    K -->|"configures"| J

    %% Monitoring Connections (dashed lines)
    B -.->|"monitors"| L
    C -.->|"monitors"| L
    D -.->|"monitors"| L
    E -.->|"monitors"| L
    F -.->|"monitors"| L
    J -.->|"monitors"| L

    %% Click Events
    click A "https://github.com/gauravssinha/petro_data_forecast/blob/main/src/utils/setup_infrastructure.py"
    click B "https://github.com/gauravssinha/petro_data_forecast/blob/main/src/data/ingest_data.py"
    click G "https://github.com/gauravssinha/petro_data_forecast/blob/main/src/models/train_model.py"
    click H "https://github.com/gauravssinha/petro_data_forecast/blob/main/src/models/scripts/model_def.py"
    click I "https://github.com/gauravssinha/petro_data_forecast/blob/main/src/models/forecast.py"
    click K "https://github.com/gauravssinha/petro_data_forecast/blob/main/src/visualization/create_dashboard.py"

    %% Styles
    classDef aws fill:#bbdefb,stroke:#0d47a1,stroke-width:2px;
    classDef custom fill:#c8e6c9,stroke:#33691e,stroke-width:2px;
```
