# ahmetzamanis/deeplearningenergyforecasting architecture

> Architecture diagram of the GitHub repository ahmetzamanis/deeplearningenergyforecasting, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ahmetzamanis/deeplearningenergyforecasting
- Repository: https://github.com/ahmetzamanis/deeplearningenergyforecasting

## Overview

##### Project Type and Purpose
- This repository is dedicated to a multi-step time series forecasting exercise on hourly energy consumption data. It employs deep learning models (a stateful LSTM and an inverted Transformer with linear extrapolation) as well as Gaussian Process Regression for analysis.
- The project supports both data analysis and model deployment. It includes Jupyter notebooks for exploratory data analysis (EDA) and modeling, source code in the src directory for model implementations, and deployment scripts (using Docker, Makefile, and Hydra configurations) to automate data ingestion, tuning, and predictions.
- Given the mix of analysis, experimentation, and automated deployment, the system design should emphasize both the research/analysis workflow and the production-level deployment pipeline.

##### Analyzing the File Structure
- Key directories include:
- “data”: Holds raw, processed, and tuning log data for analysis and deployment.
- “models”: Contains trained models and components (both for analysis and deployment).
- “notebooks”: Divided into two parts – "analysis" notebooks for data preparation, EDA, feature engineering, and model development, and "deployment" notebooks for production data workflows.
- “scripts/deployment”: Contains scripts for running batch predictions, training, tuning, and data updates, and configuration files for Hydra.
- “src”: Contains the main source code implementing deep learning modules (LSTM, Transformer, etc.), EDA functions, Gaussian Process implementations, and utilities.
- The presence of .dockerignore, Dockerfile, compose.yml, and Makefile indicates Docker-based containerization to facilitate deployment.

a. Main Components to Represent:
- Data Sources & Storage:
- External API/Data source (EPİAŞ API) as the entry point for hourly energy consumption data.
- Data directories segregated for analysis and deployment (raw, processed, logs).
- Data Ingestion / Preprocessing:
- Scripts for updating raw/training data (scripts/deployment/update_raw_data.py & update_training_data.py).
- Preprocessing modules in the “src” directory (data_extraction/api.py for API calls, deep_learning/preprocessing.py, etc.).
- Model Development and Analysis:
- Jupyter notebooks under “notebooks/analysis” for data preparation, EDA, and model experimentation.
- Source modules under “src/deep_learning” and “src/gaussian_process” representing the LSTM, Transformer, and Gaussian Process models.
- Deployment Pipeline:
- Deployment scripts under “scripts/deployment” for model training, tuning, and batch prediction.
- Hydra configurations (config.yaml) that govern script behaviors.
- Docker containerized environment defined with Dockerfile, compose.yml, and managed via Makefile.
- External Dependencies & Compute Resources:
- GPUs are used by default (noted in README via CUDA image information).
- External libraries such as PyTorch, Lightning, and GPyTorch.

b. Relationships and Interactions:
- Data flows from the external API to the “data” folder via ingestion scripts.
- Preprocessing modules consume raw data to prepare it for analysis and training.
- Analysis notebooks and “src” modules interact during model development, with outputs being written to “models” and “data” directories.
- The deployment pipeline automates model retraining/tuning and prediction tasks. Deployment scripts read configurations via Hydra, run within a Docker container, and output model artifacts and prediction results.
- A bidirectional relationship exists between source code (in “src”) and deployment scripts as changes in either reflect on how models are trained, tuned, and deployed.

c. Architectural Patterns and Design Principles:
- Separation of Concerns: There is a distinct separation between data management (data folders), model development (notebooks and src), and deployment automation (scripts, Docker, Hydra, Makefile).
- Containerization: Deployment is containerized, meaning the production environment is encapsulated in Docker images with precise dependencies.
- Modularity and Extensibility: The structure separates model implementations (deep learning vs. gaussian process) implying that adding or modifying models can be done independently.
- Reproducibility: Use of versioned configuration (Hydra) and bind mounts with Docker ensures that experiments and deployments are reproducible.

d. Relevant Technologies and Frameworks:
- PyTorch (including Lightning for model training)
- Transformers, LSTMs, and Gaussian processes for modeling
- Docker for containerizing the deployment environment
- Hydra for configuration management
- Make and compose tools for orchestrating builds and deployments

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        EP_API("EPİAŞ API"):::external
        update_raw("Update Raw Data (update_raw_data.py)"):::ingestion
        update_train("Update Training Data (update_training_data.py)"):::ingestion
        api_module("API Module (api.py)"):::preprocessing
        preprocess("Preprocessing Module (preprocessing.py)"):::preprocessing
        data_analysis("Data Storage: Analysis"):::datastorage
        data_deployment("Data Storage: Deployment"):::datastorage
    end

    %% Model Development & Analysis
    subgraph "Model Development & Analysis"
        notebooks("Analysis Notebooks (notebooks)"):::analysis
        deep_learning("Deep Learning Models (LSTM/Transformer)"):::analysis
        gaussian("Gaussian Process Models"):::analysis
        model_artifacts("Model Artifacts Storage"):::artifacts
    end

    %% Deployment Pipeline
    subgraph "Deployment Pipeline"
        deploy_scripts("Deployment Scripts (scripts/deployment)"):::deployment
        deploy_config("Deployment Configurations (config.yaml)"):::deployment
        dockerfile("Dockerfile"):::deployment
        compose("compose.yml"):::deployment
        makefile("Makefile"):::deployment
    end

    %% Data Ingestion Flow
    EP_API -->|"fetches"| update_raw
    EP_API -->|"fetches"| update_train
    update_raw --> api_module
    update_train --> api_module
    api_module --> preprocess
    preprocess --> data_analysis
    preprocess --> data_deployment

    %% Model Development Flow
    data_analysis --> notebooks
    data_analysis --> deep_learning
    data_analysis --> gaussian
    notebooks --> model_artifacts
    deep_learning --> model_artifacts
    gaussian --> model_artifacts

    %% Deployment Pipeline Flow
    data_deployment --> deploy_scripts
    deploy_config --> deploy_scripts
    dockerfile --> deploy_scripts
    compose --> deploy_scripts
    makefile --> deploy_scripts
    deploy_scripts --> model_artifacts
    deploy_scripts <--> deep_learning

    %% Click Events
    click data_analysis "https://github.com/ahmetzamanis/deeplearningenergyforecasting/tree/main/data/analysis"
    click data_deployment "https://github.com/ahmetzamanis/deeplearningenergyforecasting/tree/main/data/deployment"
    click update_raw "https://github.com/ahmetzamanis/deeplearningenergyforecasting/blob/main/scripts/deployment/update_raw_data.py"
    click update_train "https://github.com/ahmetzamanis/deeplearningenergyforecasting/blob/main/scripts/deployment/update_training_data.py"
    click api_module "https://github.com/ahmetzamanis/deeplearningenergyforecasting/blob/main/src/data_extraction/api.py"
    click preprocess "https://github.com/ahmetzamanis/deeplearningenergyforecasting/blob/main/src/deep_learning/preprocessing.py"
    click notebooks "https://github.com/ahmetzamanis/deeplearningenergyforecasting/tree/main/notebooks/analysis"
    click deep_learning "https://github.com/ahmetzamanis/deeplearningenergyforecasting/tree/main/src/deep_learning"
    click gaussian "https://github.com/ahmetzamanis/deeplearningenergyforecasting/tree/main/src/gaussian_process"
    click deploy_scripts "https://github.com/ahmetzamanis/deeplearningenergyforecasting/tree/main/scripts/deployment"
    click deploy_config "https://github.com/ahmetzamanis/deeplearningenergyforecasting/blob/main/scripts/deployment/configs/config.yaml"
    click dockerfile "https://github.com/ahmetzamanis/deeplearningenergyforecasting/tree/main/Dockerfile"
    click compose "https://github.com/ahmetzamanis/deeplearningenergyforecasting/blob/main/compose.yml"
    click makefile "https://github.com/ahmetzamanis/deeplearningenergyforecasting/tree/main/Makefile"
    click model_artifacts "https://github.com/ahmetzamanis/deeplearningenergyforecasting/tree/main/models"

    %% Styles
    classDef external fill:#ffcc99,stroke:#d35400,stroke-width:2px;
    classDef ingestion fill:#aed6f1,stroke:#2874a6,stroke-width:2px;
    classDef preprocessing fill:#abebc6,stroke:#239b56,stroke-width:2px;
    classDef datastorage fill:#fcf3cf,stroke:#f1c40f,stroke-width:2px;
    classDef analysis fill:#d7bde2,stroke:#8e44ad,stroke-width:2px;
    classDef artifacts fill:#f9e79f,stroke:#f39c12,stroke-width:2px;
    classDef deployment fill:#f5b7b1,stroke:#c0392b,stroke-width:2px;
```
