How iusztinpaul/energy-forecasting works
Main Components
• Pipeline Modules:
• Feature Pipeline: Responsible for cleaning and transforming raw data (ETL steps).
• Training Pipeline: Handles hyperparameter tuning, model training, and generating model artifacts.
• Batch Prediction Pipeline: Consumes metadata (e.g., model version from a JSON file generated by training) and serves model predictions on new data.
• Web App:
• API (FastAPI based): Serves model predictions and other insights.
• Frontend (Streamlit based): Provides a user interface for interacting with predictions.
• Monitoring (Streamlit based): Displays metrics and monitors model performance.
• Orchestration & Deployment:
• Airflow: Orchestrates the pipelines with DAGs defined under the airflow/dags directory.
• Docker & Docker Compose: Containerizes and deploys all components.
• GitHub Actions: CI/CD pipelines handle automated testing and deployment.
• External Integrations:
• Feature Store (Hopsworks): For storing and retrieving engineered features.
• ML Experiment Tracking (Weights & Biases): To log experiments, metrics, and model artifacts.
• GCP Storage: For bucket management (storing and retrieving large data or model artifacts).
• Private PyPi Server: For serving internal Python packages built via Poetry.
Out the Relationships and Data Flows
• Data Acquisition: The system ingests data from an external API (or a downloaded file simulating it) that provides hourly energy consumption data.
• Feature Engineering: The feature-pipeline module processes, cleans, and transforms the raw data before saving engineered features.
• Model Training Lifecycle: The training-pipeline reads the processed features and runs hyperparameter tuning to create and register a model; the metadata JSON files link the output across pipelines.
• Batch Predictions: The batch-prediction-pipeline reads the trained model details (via shared JSON metadata files from ML_PIPELINE_ROOT_DIR) and makes predictions on batches of data, then outputs results.
• API & Web Interface: The app-api (the FastAPI component) provides endpoints which are consumed by the frontend and monitoring modules to display predictions and system metrics.
• Orchestration: Airflow’s DAGs are used to kick off and schedule the pipelines. It ties together the building, deployment, and execution of all these steps (and connects to the private PyPi server to retrieve the packaged modules).
Architectural Patterns & Design Principles
• Modular Separation: The architecture clearly separates concerns—each pipeline module and web app component is self-contained yet interconnected through a shared metadata directory (ML_PIPELINE_ROOT_DIR).
• Containerization: Docker is used to ensure a consistent environment across different components (including orchestration with Airflow and deployment with GitHub Actions).
• CI/CD and DevOps Integration: GitHub Actions and Docker Compose are used to build, test, and deploy the system reliably.
• Microservices-Like Separation for the Web App: The API, frontend, and monitoring components are distinct services that interact via APIs.
Technologies & Tools
• Docker, Docker Compose (for containerization).
• Airflow (orchestration engine represented as a central scheduler node with DAGs).
• Python (with Poetry, used in each module’s pyproject.toml).
• FastAPI and Streamlit (in separate blocks for API, frontend, and monitoring).
• Hopsworks and Weights & Biases (external services integrated with pipelines).
• GCP (bucket storage and service account credentials) and Private PyPi Server for package management.
• GitHub Actions (CI/CD pipeline, which triggers deployments).