# iusztinpaul/energy-forecasting architecture

> Architecture diagram of the GitHub repository iusztinpaul/energy-forecasting, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/iusztinpaul/energy-forecasting
- Repository: https://github.com/iusztinpaul/energy-forecasting

## Overview

##### Main Components
- Pipeline Modules:
- Feature Pipeline: Responsible for cleaning and transforming raw data (ETL steps).
- Training Pipeline: Handles hyperparameter tuning, model training, and generating model artifacts.
- Batch Prediction Pipeline: Consumes metadata (e.g., model version from a JSON file generated by training) and serves model predictions on new data.
- Web App:
- API (FastAPI based): Serves model predictions and other insights.
- Frontend (Streamlit based): Provides a user interface for interacting with predictions.
- Monitoring (Streamlit based): Displays metrics and monitors model performance.
- Orchestration & Deployment:
- Airflow: Orchestrates the pipelines with DAGs defined under the airflow/dags directory.
- Docker & Docker Compose: Containerizes and deploys all components.
- GitHub Actions: CI/CD pipelines handle automated testing and deployment.
- External Integrations:
- Feature Store (Hopsworks): For storing and retrieving engineered features.
- ML Experiment Tracking (Weights & Biases): To log experiments, metrics, and model artifacts.
- GCP Storage: For bucket management (storing and retrieving large data or model artifacts).
- Private PyPi Server: For serving internal Python packages built via Poetry.

##### Out the Relationships and Data Flows
- Data Acquisition: The system ingests data from an external API (or a downloaded file simulating it) that provides hourly energy consumption data.
- Feature Engineering: The feature-pipeline module processes, cleans, and transforms the raw data before saving engineered features.
- Model Training Lifecycle: The training-pipeline reads the processed features and runs hyperparameter tuning to create and register a model; the metadata JSON files link the output across pipelines.
- Batch Predictions: The batch-prediction-pipeline reads the trained model details (via shared JSON metadata files from ML_PIPELINE_ROOT_DIR) and makes predictions on batches of data, then outputs results.
- API & Web Interface: The app-api (the FastAPI component) provides endpoints which are consumed by the frontend and monitoring modules to display predictions and system metrics.
- Orchestration: Airflow’s DAGs are used to kick off and schedule the pipelines. It ties together the building, deployment, and execution of all these steps (and connects to the private PyPi server to retrieve the packaged modules).

##### Architectural Patterns & Design Principles
- Modular Separation: The architecture clearly separates concerns—each pipeline module and web app component is self-contained yet interconnected through a shared metadata directory (ML_PIPELINE_ROOT_DIR).
- Containerization: Docker is used to ensure a consistent environment across different components (including orchestration with Airflow and deployment with GitHub Actions).
- CI/CD and DevOps Integration: GitHub Actions and Docker Compose are used to build, test, and deploy the system reliably.
- Microservices-Like Separation for the Web App: The API, frontend, and monitoring components are distinct services that interact via APIs.

##### Technologies & Tools
- Docker, Docker Compose (for containerization).
- Airflow (orchestration engine represented as a central scheduler node with DAGs).
- Python (with Poetry, used in each module’s pyproject.toml).
- FastAPI and Streamlit (in separate blocks for API, frontend, and monitoring).
- Hopsworks and Weights & Biases (external services integrated with pipelines).
- GCP (bucket storage and service account credentials) and Private PyPi Server for package management.
- GitHub Actions (CI/CD pipeline, which triggers deployments).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Pipelines
    subgraph "Data Pipelines"
        RD["Raw Data Source"]:::external
        FP["Feature Pipeline"]:::pipeline
        TP["Training Pipeline"]:::pipeline
        BP["Batch Prediction Pipeline"]:::pipeline
        RD -->|"ingest"| FP
        FP -->|"transform"| TP
        TP -->|"model_metadata"| BP
        FP -->|"save_features"| MS["ML_PIPELINE_ROOT_DIR"]:::container
        TP -->|"save_metadata"| MS
        BP -->|"save_predictions"| MS
    end

    %% Web Application
    subgraph "Web App"
        API["API (FastAPI)"]:::web
        FR["Frontend (Streamlit)"]:::web
        MON["Monitoring (Streamlit)"]:::web
        MS -->|"serve_data"| API
        API -->|"provides"| FR
        API -->|"provides"| MON
    end

    %% Orchestration & Containerization
    subgraph "Orchestration & Containerization"
        AIR["Airflow Orchestrator"]:::orchestrator
        DOC["Docker & Docker Compose"]:::container
        AIR -->|"triggers_feature"| FP
        AIR -->|"triggers_training"| TP
        AIR -->|"triggers_batch"| BP
        DOC -->|"containerizes"| API
        DOC -->|"containerizes"| FR
        DOC -->|"containerizes"| MON
        DOC -->|"containerizes"| AIR

        %% Containerization Files
        subgraph "Containerization Files"
            D1["app-api Dockerfile"]:::container
            D2["app-frontend Dockerfile"]:::container
            D3["app-monitoring Dockerfile"]:::container
            D4["airflow Dockerfile"]:::container
            DC["airflow/docker-compose.yaml"]:::container
            DD["deploy/app-docker-compose.yml"]:::container
        end
        DOC --> D1
        DOC --> D2
        DOC --> D3
        DOC --> D4
        DOC --> DC
        DOC --> DD
    end

    %% CI/CD & DevOps
    subgraph "CI/CD & DevOps"
        CI["GitHub Actions"]:::cicd
        CI -->|"deploys_containers"| DOC
    end

    %% External Integrations
    subgraph "External Integrations"
        HW["Hopsworks Feature Store"]:::external
        WB["Weights & Biases"]:::external
        GCP["GCP Storage"]:::external
        PP["Private PyPi Server"]:::external
        FP -->|"stores_features"| HW
        TP -->|"logs_metrics"| WB
        TP -->|"stores_artifacts"| GCP
        AIR -->|"retrieves_packages"| PP
    end

    %% Click Events
    click FP "https://github.com/iusztinpaul/energy-forecasting/tree/main/feature-pipeline/feature_pipeline"
    click TP "https://github.com/iusztinpaul/energy-forecasting/tree/main/training-pipeline/training_pipeline"
    click BP "https://github.com/iusztinpaul/energy-forecasting/tree/main/batch-prediction-pipeline/batch_prediction_pipeline"
    click API "https://github.com/iusztinpaul/energy-forecasting/tree/main/app-api/api"
    click FR "https://github.com/iusztinpaul/energy-forecasting/tree/main/app-frontend/frontend"
    click MON "https://github.com/iusztinpaul/energy-forecasting/tree/main/app-monitoring/monitoring"
    click AIR "https://github.com/iusztinpaul/energy-forecasting/tree/main/airflow/dags"
    click CI "https://github.com/iusztinpaul/energy-forecasting/tree/main/.github/workflows"
    click D1 "https://github.com/iusztinpaul/energy-forecasting/tree/main/app-api/Dockerfile"
    click D2 "https://github.com/iusztinpaul/energy-forecasting/tree/main/app-frontend/Dockerfile"
    click D3 "https://github.com/iusztinpaul/energy-forecasting/tree/main/app-monitoring/Dockerfile"
    click D4 "https://github.com/iusztinpaul/energy-forecasting/tree/main/airflow/Dockerfile"
    click DC "https://github.com/iusztinpaul/energy-forecasting/blob/main/airflow/docker-compose.yaml"
    click DD "https://github.com/iusztinpaul/energy-forecasting/blob/main/deploy/app-docker-compose.yml"

    %% Styles
    classDef pipeline fill:#add8e6,stroke:#000,stroke-width:2px;
    classDef web fill:#90ee90,stroke:#000,stroke-width:2px;
    classDef orchestrator fill:#ffa500,stroke:#000,stroke-width:2px;
    classDef cicd fill:#ffcccb,stroke:#000,stroke-width:2px;
    classDef external fill:#d3d3d3,stroke:#000,stroke-width:2px;
    classDef container fill:#ffffcc,stroke:#000,stroke-width:2px;
```
