How pyaf/load_forecasting works
Project type and purpose
• This is a full-stack data-driven web application for short-term electric load forecasting.
• The core functionality consists of:
• Data ingestion (web scraping of load and weather data)
• Model training and forecasting (time-series algorithms: ARIMA, RNN, LSTM, etc.)
• A scheduled pipeline (Celery scheduler and workers)
• A Django-based web server exposing forecasts and comparisons via HTML pages and JavaScript charts.
File structure
• Top-level folders:
• models/ – Jupyter notebooks, standalone scripts for scraping, model training, forecasting, AWS hooks
• server/ – Django project (“website” app), two Django apps (swag, users), static files, templates, Celery config
• Report/ & screenshots/ – documentation and images (not part of runtime)
• Key config files:
• server/website/settings.py – Django settings (DB, Celery broker, installed apps)
• server/requirements.txt and models/requirements.txt – Python dependencies
• celery.sh, server/celerybeat.pid – Celery starter script and beat PID
• server/db.cnf – database credentials
• Architectural patterns
• MVC: Django handles Models (ORM), Views (templates and view functions), Controllers (URL routing)
• Task queue: Celery + broker (Redis/RabbitMQ) + beat scheduler
• Modular data pipeline: separate scripts for scraping, hyperparameter search, training, forecasting
README for additional insights
• Confirms project workflow:
1. Scrape load/weather data from external sites
2. Store raw data (CSVs) and/or in database
3. Run forecasting scripts daily at 00:30 IST via AWS scheduler or Celery
4. Serve results via Django web server, visualized with c3.js and jQuery
Elements
• Clear component labels (e.g., “Load Scraper Service”, “Forecasting Worker”)
• External systems
• Data repositories
• Compute services
• Front-end clients