akshat-rai0/ntpc-nvvn-mp

1. This repository is a Django-based forecasting system centered on a state-specific prediction pipeline. The runtime is split into three main Django apps: states for domain data, predictions for ingestion/inference/API, and dashboard for the human-facing UI. The project package demand_predictor/ wires these apps together through Django settings, URL routing, and WSGI/ASGI entrypoints.

2. The core business object is “state.” Each state is configured in config/states/*.yaml with a short code, MERIT India demand URL, model path, timezone, fallback demand, and a set of cities with weather weights. That YAML drives registration in the database and determines which external source, model file, and weather locations are used for that state.

3. states/ defines the persistent domain layer. It holds the Django models and admin registration for state metadata and time-series records such as live demand readings and prediction records. This is the main boundary between the business data model and everything else in the system.

4. predictions/services/ is the engine room. merit_client.py fetches live demand from MERIT India, weather.py in utils/ fetches weather from Open-Meteo, time_features.py and lag_store.py derive model inputs, predictor.py runs single-step inference, and forecaster.py handles day-ahead/multi-slot forecasts. registry.py loads YAML state definitions, while csv_sync.py mirrors database changes into CSV files.

5. The model boundary is explicit and file-based. Each state has a trained LightGBM model stored under models/{state_code}/lgbm_final.txt or, for a few states, a pickle artifact. Inference expects a fixed feature vector assembled from weather, calendar, and lagged demand inputs; the system is therefore tightly coupled to these model assets and their feature order.

6. Data flows from external services into the pipeline, then into storage and visualization. On each refresh, the app pulls live demand and weather, computes features, performs prediction, saves both actual and predicted values to SQLite, and syncs them into CSV logs under data/states/{code}/. Historical CSVs in data/ are also used to backfill lag features and bootstrap new states.

7. The scheduling boundary is handled inside predictions/scheduler.py and app startup logic. A background job refreshes active states every five minutes when enabled, so the system can operate continuously without manual intervention. The same refresh workflow is also exposed as a management command for manual execution and operational control.

8. The public API sits in predictions/api/views.py and predictions/urls.py. It exposes state listing plus per-state “today,” “tomorrow,” “forecast,” and “history” endpoints, returning JSON for consumption by the dashboard or external systems. This API reads from the same persistence layer used by the scheduler, so it is not a separate data source.

9. The dashboard in dashboard/ is a presentation layer over the API/database. templates/dashboard/index.html and the static Chart.js frontend render actual vs predicted demand, forecast views, and summary metrics. It depends on the prediction data being current, but it does not own the forecasting logic itself.

10. Persistence is dual-layered. SQLite (db.sqlite3) is the primary system-of-record for Django models, while CSV exports provide portability, backup, and offline analysis. The CSV layer is not authoritative; it is a synchronized projection of database state.

11. Operational tooling is first-class. Management commands under predictions/management/commands/ support seeding state definitions, refreshing demand, and importing historical demand. The standalone prediction.py script is a lightweight CLI entrypoint for running a single-state prediction without the full server stack.

12. The repository also contains a substantial data/science workspace under data/ with notebooks and intermediate CSVs. This area is separate from the application runtime: it supports model development, feature engineering, and data preparation, while the deployed app consumes only the curated model files, YAML configs, and runtime logs.

13. In architectural terms, the main dependency direction is: external services and historical data → feature engineering and model inference → SQLite/CSV persistence → API and dashboard. The key boundaries are between state configuration, prediction services, persistence, and presentation, with Django acting as the integration framework across all of them.