How ayushd64/high_dim_demand_forecasting-time-series- works
1. Repository shape
- This repo is organized as a notebook-driven forecasting pipeline plus a separate Streamlit inference app.
- The operational split is:
- notebooks/walmart_M5_forecasting.ipynb for end-to-end data prep, training, and evaluation
- src/ingestion.py for reusable ingestion/memory-optimization logic
- app.py for interactive model serving
- The README mentions additional config/, features/, models/, and evaluation/ packages, but those are not present in the provided tree.
2. External data boundary
- The system consumes the Walmart M5 Forecasting dataset via Kaggle API.
- Raw data arrives in the competition’s wide panel format: one row per item-store series, many day columns, plus calendar and price-related tables.
- This is the main external data ingress point and the first architectural boundary.
3. Ingestion and normalization layer
- src/ingestion.py is the main utility module.
- Its role is to transform the raw wide-format retail tables into a long/tidy sales panel with explicit time keys such as (id, date, demand).
- It also performs column downcasting and categorical conversion to reduce memory footprint, which is a critical enabler for handling the full M5 scale on a single machine.
4. Feature engineering boundary
- The notebook builds the supervised-learning matrix from the long-format panel.
- Feature families called out in the README include:
- lag features like lag_7, lag_14, lag_21, lag_28
- rolling statistics such as means and standard deviations
- calendar, event, and price covariates
- target/item popularity encodings
- Conceptually, this layer converts a time-series panel into a tabular regression problem.
5. Modeling core
- The central predictor is a single global LightGBM model trained across all item-store-time tuples.
- This is a key architectural choice: rather than one model per series, the system pools cross-series signal into one shared learner.
- The model is trained on the engineered tabular matrix and saved as a serialized artifact.
6. Temporal validation boundary
- Evaluation uses a strict, non-shuffled 28-day holdout window.
- This preserves time order and prevents look-ahead leakage.
- For an architecture diagram, this is an important data-flow split: training horizon on one side, final 28-day validation on the other.
7. Forecast evaluation and decision layer
- The README describes two downstream evaluation/decision concerns:
- RMSE for forecast accuracy
- an asymmetric inventory loss / NEV engine using different holding vs. stockout costs
- This means the forecast is not just scored statistically; it is also mapped into an inventory decision policy with business cost asymmetry.
- The safety-stock simulation and capacity ceiling checks are part of this operational decision layer.
8. Serving / deployment layer
- app.py is the runtime interface for interactive use.
- It loads a serialized model artifact from the notebook outputs and exposes demand forecasting through a Streamlit UI.
- The UI is meant for scenario simulation, inventory recommendation, and forecast visualization rather than batch training.
9. Artifact flow
- The main persistent artifact is the trained model pickle referenced in the README as living under notebooks/.
- Data flow is therefore:
- raw Kaggle data → ingestion/cleanup → engineered feature table → LightGBM training → serialized model artifact → Streamlit inference
- Intermediate cleaned panels are likely materialized under data/processed/ per the README, though that directory is not shown in the tree.
10. Tooling and runtime dependencies
- The repo is built around the Python data stack, with practical dependence on:
- pandas/numpy for transformation
- LightGBM for training
- joblib for model serialization
- Streamlit and Plotly for the app
- Jupyter for orchestration and experimentation
- Kaggle API for dataset acquisition
- These choices matter architecturally because the pipeline is designed to run end-to-end on a workstation, not as a distributed service.
11. What to draw as the main subsystems
- A good diagram would show four boundaries:
- External data source: Kaggle M5 dataset
- Preparation layer: ingestion + downcasting + long-format transformation
- Modeling layer: feature engineering + global LightGBM + temporal validation
- Consumption layer: Streamlit app reading the saved model and producing forecasts/inventory guidance
12. Overall architectural character
- This is a single-machine analytical ML system, not a microservice or distributed platform.
- The notebook is the orchestration center, src/ingestion.py is the reusable preprocessing component, and app.py is the thin serving surface over a persisted forecast model.
Main entry points: walmart_M5_forecasting.ipynb Forecast notebook · ingestion.py Ingestion · app.py Streamlit app · 01_demand_forecasting.ipynb Demand notebook
Components 17
External data
Source boundary for the M5 retail dataset
- Kaggle M5 · External Walmart M5 Forecasting data source consumed through the Kaggle API
- Wide tables · Competition-format tables with item-store series and day columns
Preparation
Notebook-led ingestion and panel normalization
- Forecast notebook · End-to-end pipeline for data prep, training, and evaluation
notebooks/walmart_M5_forecasting.ipynb - Demand notebook · Additional notebook present in the repo for demand forecasting experiments
notebooks/01_demand_forecasting.ipynb - Ingestion · Reusable raw-table ingestion, reshaping, and memory optimization logic
src/ingestion.py - Long panel · Normalized time-indexed sales table with explicit date and demand fields
- Downcasting · Column downcasting and categorical conversion to fit M5-scale data on one machine
Modeling
Feature engineering, training, and validation
- Feature build · Builds supervised learning rows from the time-series panel
- Lag features · Lagged demand signals such as 7, 14, 21, and 28 day offsets
- Rolling stats · Rolling means and standard deviations over recent history
- Calendar covariates · Calendar, event, and price-related predictors
- Popularity encodings · Target and item popularity encodings used to enrich the learner
- 28-day holdout · Strict non-shuffled validation window preserving time order
- LightGBM model · Single shared model trained across all item-store-time tuples
Serving
Streamlit inference and decision support
- Model artifact · Persisted trained model loaded for later inference
notebooks/walmart_M5_forecasting.ipynb - Streamlit app · Interactive forecasting, scenario simulation, and visualization surface
app.py - Inventory policy · Maps forecasts into asymmetric cost, safety-stock, and capacity guidance
Connections 24
- Kaggle M5 → Wide tables: downloads · Kaggle API retrieves the raw competition tables
- Wide tables → Ingestion: ingests · Raw wide-format tables are passed into the reusable ingestion layer
- Ingestion → Downcasting: optimizes · Numeric and categorical columns are reduced for memory efficiency
- Ingestion → Long panel: reshapes · Wide retail tables are converted into a long, time-indexed sales panel
- Downcasting → Long panel: produces · Compact panel representation ready for feature generation
- Long panel → Feature build: inputs · The tidy sales panel is the base table for supervised learning
- Feature build → Lag features: adds · Historical demand lags are appended to each training row
- Feature build → Rolling stats: adds · Rolling window summary statistics are computed from recent demand
- Feature build → Calendar covariates: adds · Calendar, event, and price signals are merged in
- Feature build → Popularity encodings: adds · Series and item popularity information is encoded
- Lag features → 28-day holdout: enters · Engineered lag signals are evaluated under temporal holdout
- Rolling stats → 28-day holdout: enters · Rolling features are validated on a future window
- Calendar covariates → 28-day holdout: enters · Exogenous features participate in the split
- Popularity encodings → 28-day holdout: enters · Cross-series encodings are checked against the holdout
- 28-day holdout → LightGBM model: trains · A global LightGBM learner is fit on the training horizon
- LightGBM model → Model artifact: serializes · The trained model is persisted as a reusable artifact
- Model artifact → Streamlit app: loads · Streamlit runtime loads the serialized model for inference
- Streamlit app → Inventory policy: simulates · Forecasts are translated into inventory recommendations and cost tradeoffs
- LightGBM model → Inventory policy: scores · Predictions feed RMSE and asymmetric inventory-loss evaluation
- Forecast notebook → Ingestion: orchestrates · Notebook calls the preprocessing module during the end-to-end pipeline
- Forecast notebook → Feature build: builds · Notebook drives feature matrix construction
- Forecast notebook → LightGBM model: trains · Notebook runs model fitting and persists the result
- Forecast notebook → Inventory policy: evaluates · Notebook computes validation metrics and business-cost checks
- Demand notebook → Feature build: explores · Secondary notebook supports demand forecasting experimentation