# pyaf/load_forecasting architecture

> Architecture diagram of the GitHub repository pyaf/load_forecasting, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pyaf/load_forecasting
- Repository: https://github.com/pyaf/load_forecasting

## Overview

##### Project type and purpose
- This is a full-stack data-driven web application for short-term electric load forecasting.
- The core functionality consists of:
- Data ingestion (web scraping of load and weather data)
- Model training and forecasting (time-series algorithms: ARIMA, RNN, LSTM, etc.)
- A scheduled pipeline (Celery scheduler and workers)
- A Django-based web server exposing forecasts and comparisons via HTML pages and JavaScript charts.

##### File structure
- Top-level folders:
- models/ – Jupyter notebooks, standalone scripts for scraping, model training, forecasting, AWS hooks
- server/ – Django project (“website” app), two Django apps (swag, users), static files, templates, Celery config
- Report/ & screenshots/ – documentation and images (not part of runtime)
- Key config files:
- server/website/settings.py – Django settings (DB, Celery broker, installed apps)
- server/requirements.txt and models/requirements.txt – Python dependencies
- celery.sh, server/celerybeat.pid – Celery starter script and beat PID
- server/db.cnf – database credentials
- Architectural patterns
- MVC: Django handles Models (ORM), Views (templates and view functions), Controllers (URL routing)
- Task queue: Celery + broker (Redis/RabbitMQ) + beat scheduler
- Modular data pipeline: separate scripts for scraping, hyperparameter search, training, forecasting

##### README for additional insights
- Confirms project workflow:
1. Scrape load/weather data from external sites
2. Store raw data (CSVs) and/or in database
3. Run forecasting scripts daily at 00:30 IST via AWS scheduler or Celery
4. Serve results via Django web server, visualized with c3.js and jQuery

##### Elements
- Clear component labels (e.g., “Load Scraper Service”, “Forecasting Worker”)
- External systems
- Data repositories
- Compute services
- Front-end clients

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    External1["SLDC Website"]:::external
    External2["Weather Underground"]:::external

    %% Data Ingestion Pipeline
    subgraph "Data Ingestion"   
        Scraper1["Load Scraper Service"]:::pipeline
        Scraper2["Weather Scraper Service"]:::pipeline
    end

    %% Data Storage
    subgraph "Data Storage"
        StorageCSV["Local CSV Logs"]:::storage
        DB["Relational Database"]:::storage
    end

    %% Scheduler and Task Queue
    subgraph "Scheduler & Task Queue"
        CeleryStart["Celery Startup Script"]:::compute
        Scheduler["Celery Beat Scheduler"]:::compute
        CeleryPID["Celery Beat PID File"]:::compute
        Broker["Redis/RabbitMQ Broker"]:::compute
        Worker["Celery Workers"]:::compute
    end

    %% Model Services
    subgraph "Model Services"
        Utils["Shared Model Utilities"]:::pipeline
        Notebooks["Jupyter Notebook Prototypes"]:::pipeline
        Model1["ARIMA Hyperparameter Search"]:::pipeline
        Model2["ARIMA Forecasting Worker"]:::pipeline
        Model3["RNN Forecasting Worker"]:::pipeline
        Model4["Exponential Smoothing Worker"]:::pipeline
    end

    %% Django Backend
    subgraph "Django Backend"
        Settings["Django Settings & Config"]:::backend
        URLs["Django URL Routing"]:::backend
        WSGI["Django WSGI Entry Point"]:::backend
        Manage["Django Management Utility"]:::backend

        subgraph "App: swag"
            SwagViews["Views & Data Loader"]:::backend
            SwagTasks["Celery Task Definitions"]:::backend
            SwagModelsAdmin["Models & Admin"]:::backend
            SwagForms["Forms"]:::backend
            SwagMigrations["Migrations"]:::backend
            SwagTests["Tests"]:::backend
        end

        subgraph "App: users"
            UsersViews["Views & Models"]:::backend
            UsersAdmin["Admin & App Config"]:::backend
            UsersMigrations["Migrations"]:::backend
            UsersTests["Tests"]:::backend
        end

        ConfigCelery["Celery App Config"]:::backend
        DBConfig["Database Credentials"]:::backend
        ReqServer["Python Dependencies (Web Server)"]:::backend
    end

    %% Frontend
    subgraph "Frontend"
        Templates["HTML Templates"]:::frontend
        StaticAssets["Static Assets (c3.js, jQuery, main.js)"]:::frontend
    end

    %% Client
    Client["Client Browser"]:::frontend

    %% Connections
    External1 -->|scrapes data| Scraper1
    External2 -->|scrapes data| Scraper2
    Scraper1 -->|writes CSV| StorageCSV
    Scraper2 -->|writes CSV| StorageCSV

    CeleryStart --> Scheduler
    Scheduler --> CeleryPID
    Scheduler -.->|triggers tasks| Worker
    Worker -->|reads tasks| Model1
    Worker -->|reads tasks| Model2
    Worker -->|reads tasks| Model3
    Worker -->|reads tasks| Model4

    StorageCSV -->|read CSV| Model1
    StorageCSV -->|read CSV| Model2
    StorageCSV -->|read CSV| Model3
    StorageCSV -->|read CSV| Model4

    Utils -->|used by| Model1
    Utils -->|used by| Model2
    Utils -->|used by| Model3
    Utils -->|used by| Model4
    Notebooks -->|prototypes for| Model1
    Notebooks -->|prototypes for| Model2
    Notebooks -->|prototypes for| Model3
    Notebooks -->|prototypes for| Model4

    Model1 -->|writes forecast| DB
    Model2 -->|writes forecast| DB
    Model3 -->|writes forecast| DB
    Model4 -->|writes forecast| DB

    DB -->|read| Settings
    DB -->|read/write| SwagViews

    Settings --> URLs
    Settings --> ConfigCelery
    URLs --> WSGI
    WSGI --> Manage

    ConfigCelery -->|initializes| Worker
    SwagTasks -->|defines| Worker

    SwagViews -->|renders| Templates
    SwagViews -->|serves JSON| StaticAssets
    UsersViews -->|renders| Templates

    Templates -->|delivered to| Client
    StaticAssets -->|loaded by| Client

    %% Click Events
    click Scraper1 "https://github.com/pyaf/load_forecasting/blob/master/models/load_scrap.py"
    click Scraper2 "https://github.com/pyaf/load_forecasting/blob/master/models/whether_scrap.py"
    click Model1 "https://github.com/pyaf/load_forecasting/blob/master/models/pdq_search.py"
    click Model2 "https://github.com/pyaf/load_forecasting/blob/master/models/aws_arima.py"
    click Model3 "https://github.com/pyaf/load_forecasting/blob/master/models/aws_rnn.py"
    click Model4 "https://github.com/pyaf/load_forecasting/blob/master/models/aws_smoothing.py"
    click Utils "https://github.com/pyaf/load_forecasting/blob/master/models/utils.py"
    click Notebooks "https://github.com/pyaf/load_forecasting/tree/master/models/"
    click StorageCSV "https://github.com/pyaf/load_forecasting/tree/master/models/log/"
    click CeleryStart "https://github.com/pyaf/load_forecasting/blob/master/celery.sh"
    click CeleryPID "https://github.com/pyaf/load_forecasting/blob/master/server/celerybeat.pid"
    click ConfigCelery "https://github.com/pyaf/load_forecasting/blob/master/server/website/for_celery.py"
    click SwagTasks "https://github.com/pyaf/load_forecasting/blob/master/server/swag/tasks.py"
    click Settings "https://github.com/pyaf/load_forecasting/blob/master/server/website/settings.py"
    click URLs "https://github.com/pyaf/load_forecasting/blob/master/server/website/urls.py"
    click WSGI "https://github.com/pyaf/load_forecasting/blob/master/server/website/wsgi.py"
    click Manage "https://github.com/pyaf/load_forecasting/blob/master/server/manage.py"
    click SwagViews "https://github.com/pyaf/load_forecasting/blob/master/server/swag/views.py"
    click SwagModelsAdmin "https://github.com/pyaf/load_forecasting/blob/master/server/swag/models.py"
    click SwagModelsAdmin "https://github.com/pyaf/load_forecasting/blob/master/server/swag/admin.py"
    click SwagForms "https://github.com/pyaf/load_forecasting/blob/master/server/swag/forms.py"
    click SwagMigrations "https://github.com/pyaf/load_forecasting/tree/master/server/swag/migrations/"
    click SwagTests "https://github.com/pyaf/load_forecasting/blob/master/server/swag/tests.py"
    click UsersViews "https://github.com/pyaf/load_forecasting/blob/master/server/users/views.py"
    click UsersViews "https://github.com/pyaf/load_forecasting/blob/master/server/users/models.py"
    click UsersAdmin "https://github.com/pyaf/load_forecasting/blob/master/server/users/admin.py"
    click UsersAdmin "https://github.com/pyaf/load_forecasting/blob/master/server/users/apps.py"
    click UsersMigrations "https://github.com/pyaf/load_forecasting/tree/master/server/users/migrations/"
    click UsersTests "https://github.com/pyaf/load_forecasting/blob/master/server/users/tests.py"
    click StaticAssets "https://github.com/pyaf/load_forecasting/tree/master/server/static/"
    click Templates "https://github.com/pyaf/load_forecasting/tree/master/server/tamplates/"
    click DBConfig "https://github.com/pyaf/load_forecasting/blob/master/server/db.cnf"
    click ReqServer "https://github.com/pyaf/load_forecasting/blob/master/server/requirements.txt"
    click ReqModels "https://github.com/pyaf/load_forecasting/blob/master/models/requirements.txt"

    %% Styles
    classDef external fill:#f9f,stroke:#333,stroke-width:2px
    classDef pipeline fill:#ffb,stroke:#333,stroke-width:2px
    classDef storage fill:#bbf,stroke:#333,stroke-width:2px
    classDef compute fill:#fbf,stroke:#333,stroke-width:2px
    classDef backend fill:#bfb,stroke:#333,stroke-width:2px
    classDef frontend fill:#fbb,stroke:#333,stroke-width:2px
```
