How ahmetzamanis/deeplearningenergyforecasting works
Project Type and Purpose
• This repository is dedicated to a multi-step time series forecasting exercise on hourly energy consumption data. It employs deep learning models (a stateful LSTM and an inverted Transformer with linear extrapolation) as well as Gaussian Process Regression for analysis.
• The project supports both data analysis and model deployment. It includes Jupyter notebooks for exploratory data analysis (EDA) and modeling, source code in the src directory for model implementations, and deployment scripts (using Docker, Makefile, and Hydra configurations) to automate data ingestion, tuning, and predictions.
• Given the mix of analysis, experimentation, and automated deployment, the system design should emphasize both the research/analysis workflow and the production-level deployment pipeline.
Analyzing the File Structure
• Key directories include:
• “data”: Holds raw, processed, and tuning log data for analysis and deployment.
• “models”: Contains trained models and components (both for analysis and deployment).
• “notebooks”: Divided into two parts – "analysis" notebooks for data preparation, EDA, feature engineering, and model development, and "deployment" notebooks for production data workflows.
• “scripts/deployment”: Contains scripts for running batch predictions, training, tuning, and data updates, and configuration files for Hydra.
• “src”: Contains the main source code implementing deep learning modules (LSTM, Transformer, etc.), EDA functions, Gaussian Process implementations, and utilities.
• The presence of .dockerignore, Dockerfile, compose.yml, and Makefile indicates Docker-based containerization to facilitate deployment.
a. Main Components to Represent:
• Data Sources & Storage:
• External API/Data source (EPİAŞ API) as the entry point for hourly energy consumption data.
• Data directories segregated for analysis and deployment (raw, processed, logs).
• Data Ingestion / Preprocessing:
• Scripts for updating raw/training data (scripts/deployment/update_raw_data.py & update_training_data.py).
• Preprocessing modules in the “src” directory (data_extraction/api.py for API calls, deep_learning/preprocessing.py, etc.).
• Model Development and Analysis:
• Jupyter notebooks under “notebooks/analysis” for data preparation, EDA, and model experimentation.
• Source modules under “src/deep_learning” and “src/gaussian_process” representing the LSTM, Transformer, and Gaussian Process models.
• Deployment Pipeline:
• Deployment scripts under “scripts/deployment” for model training, tuning, and batch prediction.
• Hydra configurations (config.yaml) that govern script behaviors.
• Docker containerized environment defined with Dockerfile, compose.yml, and managed via Makefile.
• External Dependencies & Compute Resources:
• GPUs are used by default (noted in README via CUDA image information).
• External libraries such as PyTorch, Lightning, and GPyTorch.
b. Relationships and Interactions:
• Data flows from the external API to the “data” folder via ingestion scripts.
• Preprocessing modules consume raw data to prepare it for analysis and training.
• Analysis notebooks and “src” modules interact during model development, with outputs being written to “models” and “data” directories.
• The deployment pipeline automates model retraining/tuning and prediction tasks. Deployment scripts read configurations via Hydra, run within a Docker container, and output model artifacts and prediction results.
• A bidirectional relationship exists between source code (in “src”) and deployment scripts as changes in either reflect on how models are trained, tuned, and deployed.
c. Architectural Patterns and Design Principles:
• Separation of Concerns: There is a distinct separation between data management (data folders), model development (notebooks and src), and deployment automation (scripts, Docker, Hydra, Makefile).
• Containerization: Deployment is containerized, meaning the production environment is encapsulated in Docker images with precise dependencies.
• Modularity and Extensibility: The structure separates model implementations (deep learning vs. gaussian process) implying that adding or modifying models can be done independently.
• Reproducibility: Use of versioned configuration (Hydra) and bind mounts with Docker ensures that experiments and deployments are reproducible.
d. Relevant Technologies and Frameworks:
• PyTorch (including Lightning for model training)
• Transformers, LSTMs, and Gaussian processes for modeling
• Docker for containerizing the deployment environment
• Hydra for configuration management
• Make and compose tools for orchestrating builds and deployments