# ayushd64/high_dim_demand_forecasting-time-series- architecture

> Architecture diagram of the GitHub repository ayushd64/high_dim_demand_forecasting-time-series-, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayushd64/high_dim_demand_forecasting-time-series-
- Repository: https://github.com/ayushd64/high_dim_demand_forecasting-time-series-

## Overview

1. **Repository shape**
   - This repo is organized as a **notebook-driven forecasting pipeline** plus a separate **Streamlit inference app**.
   - The operational split is:
     - `notebooks/walmart_M5_forecasting.ipynb` for end-to-end data prep, training, and evaluation
     - `src/ingestion.py` for reusable ingestion/memory-optimization logic
     - `app.py` for interactive model serving
   - The README mentions additional `config/`, `features/`, `models/`, and `evaluation/` packages, but those are not present in the provided tree.

2. **External data boundary**
   - The system consumes the **Walmart M5 Forecasting dataset** via **Kaggle API**.
   - Raw data arrives in the competition’s **wide panel format**: one row per item-store series, many day columns, plus calendar and price-related tables.
   - This is the main external data ingress point and the first architectural boundary.

3. **Ingestion and normalization layer**
   - `src/ingestion.py` is the main utility module.
   - Its role is to transform the raw wide-format retail tables into a **long/tidy sales panel** with explicit time keys such as `(id, date, demand)`.
   - It also performs **column downcasting** and categorical conversion to reduce memory footprint, which is a critical enabler for handling the full M5 scale on a single machine.

4. **Feature engineering boundary**
   - The notebook builds the supervised-learning matrix from the long-format panel.
   - Feature families called out in the README include:
     - **lag features** like `lag_7`, `lag_14`, `lag_21`, `lag_28`
     - **rolling statistics** such as means and standard deviations
     - **calendar, event, and price covariates**
     - **target/item popularity encodings**
   - Conceptually, this layer converts a time-series panel into a tabular regression problem.

5. **Modeling core**
   - The central predictor is a **single global LightGBM model** trained across all item-store-time tuples.
   - This is a key architectural choice: rather than one model per series, the system pools cross-series signal into one shared learner.
   - The model is trained on the engineered tabular matrix and saved as a serialized artifact.

6. **Temporal validation boundary**
   - Evaluation uses a **strict, non-shuffled 28-day holdout window**.
   - This preserves time order and prevents look-ahead leakage.
   - For an architecture diagram, this is an important data-flow split: training horizon on one side, final 28-day validation on the other.

7. **Forecast evaluation and decision layer**
   - The README describes two downstream evaluation/decision concerns:
     - **RMSE** for forecast accuracy
     - an **asymmetric inventory loss / NEV engine** using different holding vs. stockout costs
   - This means the forecast is not just scored statistically; it is also mapped into an inventory decision policy with business cost asymmetry.
   - The safety-stock simulation and capacity ceiling checks are part of this operational decision layer.

8. **Serving / deployment layer**
   - `app.py` is the runtime interface for interactive use.
   - It loads a serialized model artifact from the notebook outputs and exposes demand forecasting through a **Streamlit** UI.
   - The UI is meant for scenario simulation, inventory recommendation, and forecast visualization rather than batch training.

9. **Artifact flow**
   - The main persistent artifact is the trained model pickle referenced in the README as living under `notebooks/`.
   - Data flow is therefore:
     - raw Kaggle data → ingestion/cleanup → engineered feature table → LightGBM training → serialized model artifact → Streamlit inference
   - Intermediate cleaned panels are likely materialized under `data/processed/` per the README, though that directory is not shown in the tree.

10. **Tooling and runtime dependencies**
    - The repo is built around the Python data stack, with practical dependence on:
      - **pandas/numpy** for transformation
      - **LightGBM** for training
      - **joblib** for model serialization
      - **Streamlit** and **Plotly** for the app
      - **Jupyter** for orchestration and experimentation
      - **Kaggle API** for dataset acquisition
    - These choices matter architecturally because the pipeline is designed to run end-to-end on a workstation, not as a distributed service.

11. **What to draw as the main subsystems**
    - A good diagram would show four boundaries:
      - **External data source**: Kaggle M5 dataset
      - **Preparation layer**: ingestion + downcasting + long-format transformation
      - **Modeling layer**: feature engineering + global LightGBM + temporal validation
      - **Consumption layer**: Streamlit app reading the saved model and producing forecasts/inventory guidance

12. **Overall architectural character**
    - This is a **single-machine analytical ML system**, not a microservice or distributed platform.
    - The notebook is the orchestration center, `src/ingestion.py` is the reusable preprocessing component, and `app.py` is the thin serving surface over a persisted forecast model.

Main entry points: [`notebooks/walmart_M5_forecasting.ipynb`](https://github.com/ayushd64/high_dim_demand_forecasting-time-series-/blob/main/notebooks/walmart_M5_forecasting.ipynb) (Forecast notebook), [`src/ingestion.py`](https://github.com/ayushd64/high_dim_demand_forecasting-time-series-/blob/main/src/ingestion.py) (Ingestion), [`app.py`](https://github.com/ayushd64/high_dim_demand_forecasting-time-series-/blob/main/app.py) (Streamlit app), [`notebooks/01_demand_forecasting.ipynb`](https://github.com/ayushd64/high_dim_demand_forecasting-time-series-/blob/main/notebooks/01_demand_forecasting.ipynb) (Demand notebook)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_external["External data"]
  node_kaggle_m5[("Kaggle M5<br/>dataset source")]
  node_wide_raw[("Wide tables<br/>raw retail panel")]
end

subgraph group_prep["Preparation"]
  node_notebook_main["Forecast notebook<br/>orchestration notebook"]
  node_notebook_alt["Demand notebook<br/>exploratory notebook"]
  node_ingestion["Ingestion<br/>preprocessing module<br/>[ingestion.py]"]
  node_long_panel[("Long panel<br/>tidy sales panel")]
  node_downcast["Downcasting<br/>memory optimization"]
end

subgraph group_modeling["Modeling"]
  node_features["Feature build<br/>tabular featurizer"]
  node_lags["Lag features<br/>temporal features"]
  node_rolling["Rolling stats<br/>aggregate features"]
  node_covariates["Calendar covariates<br/>exogenous features"]
  node_target_enc["Popularity encodings<br/>cross-series features"]
  node_train_split["28-day holdout<br/>temporal validation"]
  node_lgbm["LightGBM model<br/>global predictor"]
end

subgraph group_serving["Serving"]
  node_artifact[("Model artifact<br/>serialized model")]
  node_app["Streamlit app<br/>inference ui<br/>[app.py]"]
  node_decision["Inventory policy<br/>decision layer"]
end

node_kaggle_m5 -->|"downloads"| node_wide_raw
node_wide_raw -->|"ingests"| node_ingestion
node_ingestion -->|"optimizes"| node_downcast
node_ingestion -->|"reshapes"| node_long_panel
node_downcast -->|"produces"| node_long_panel
node_long_panel -->|"inputs"| node_features
node_features -->|"adds"| node_lags
node_features -->|"adds"| node_rolling
node_features -->|"adds"| node_covariates
node_features -->|"adds"| node_target_enc
node_lags -->|"enters"| node_train_split
node_rolling -->|"enters"| node_train_split
node_covariates -->|"enters"| node_train_split
node_target_enc -->|"enters"| node_train_split
node_train_split -->|"trains"| node_lgbm
node_lgbm -->|"serializes"| node_artifact
node_artifact -->|"loads"| node_app
node_app -->|"simulates"| node_decision
node_lgbm -->|"scores"| node_decision
node_notebook_main -->|"orchestrates"| node_ingestion
node_notebook_main -->|"builds"| node_features
node_notebook_main -->|"trains"| node_lgbm
node_notebook_main -->|"evaluates"| node_decision
node_notebook_alt -.->|"explores"| node_features

click node_notebook_main "https://github.com/ayushd64/high_dim_demand_forecasting-time-series-/blob/main/notebooks/walmart_M5_forecasting.ipynb"
click node_notebook_alt "https://github.com/ayushd64/high_dim_demand_forecasting-time-series-/blob/main/notebooks/01_demand_forecasting.ipynb"
click node_ingestion "https://github.com/ayushd64/high_dim_demand_forecasting-time-series-/blob/main/src/ingestion.py"
click node_artifact "https://github.com/ayushd64/high_dim_demand_forecasting-time-series-/blob/main/notebooks/walmart_M5_forecasting.ipynb"
click node_app "https://github.com/ayushd64/high_dim_demand_forecasting-time-series-/blob/main/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_kaggle_m5,node_wide_raw toneBlue
class node_notebook_main,node_notebook_alt,node_ingestion,node_long_panel,node_downcast toneAmber
class node_features,node_lags,node_rolling,node_covariates,node_target_enc,node_train_split,node_lgbm toneMint
class node_artifact,node_app,node_decision toneRose
```

## Components

### External data

Source boundary for the M5 retail dataset

- **Kaggle M5**: dataset source · External Walmart M5 Forecasting data source consumed through the Kaggle API
- **Wide tables**: raw retail panel · Competition-format tables with item-store series and day columns

### Preparation

Notebook-led ingestion and panel normalization

- [Forecast notebook](https://github.com/ayushd64/high_dim_demand_forecasting-time-series-/blob/main/notebooks/walmart_M5_forecasting.ipynb): orchestration notebook · End-to-end pipeline for data prep, training, and evaluation · `notebooks/walmart_M5_forecasting.ipynb`
- [Demand notebook](https://github.com/ayushd64/high_dim_demand_forecasting-time-series-/blob/main/notebooks/01_demand_forecasting.ipynb): exploratory notebook · Additional notebook present in the repo for demand forecasting experiments · `notebooks/01_demand_forecasting.ipynb`
- [Ingestion](https://github.com/ayushd64/high_dim_demand_forecasting-time-series-/blob/main/src/ingestion.py): preprocessing module · Reusable raw-table ingestion, reshaping, and memory optimization logic · `src/ingestion.py`
- **Long panel**: tidy sales panel · Normalized time-indexed sales table with explicit date and demand fields
- **Downcasting**: memory optimization · Column downcasting and categorical conversion to fit M5-scale data on one machine

### Modeling

Feature engineering, training, and validation

- **Feature build**: tabular featurizer · Builds supervised learning rows from the time-series panel
- **Lag features**: temporal features · Lagged demand signals such as 7, 14, 21, and 28 day offsets
- **Rolling stats**: aggregate features · Rolling means and standard deviations over recent history
- **Calendar covariates**: exogenous features · Calendar, event, and price-related predictors
- **Popularity encodings**: cross-series features · Target and item popularity encodings used to enrich the learner
- **28-day holdout**: temporal validation · Strict non-shuffled validation window preserving time order
- **LightGBM model**: global predictor · Single shared model trained across all item-store-time tuples

### Serving

Streamlit inference and decision support

- [Model artifact](https://github.com/ayushd64/high_dim_demand_forecasting-time-series-/blob/main/notebooks/walmart_M5_forecasting.ipynb): serialized model · Persisted trained model loaded for later inference · `notebooks/walmart_M5_forecasting.ipynb`
- [Streamlit app](https://github.com/ayushd64/high_dim_demand_forecasting-time-series-/blob/main/app.py): inference ui · Interactive forecasting, scenario simulation, and visualization surface · `app.py`
- **Inventory policy**: decision layer · Maps forecasts into asymmetric cost, safety-stock, and capacity guidance

## Connections

- Kaggle M5 → Wide tables: downloads · Kaggle API retrieves the raw competition tables
- Wide tables → Ingestion: ingests · Raw wide-format tables are passed into the reusable ingestion layer
- Ingestion → Downcasting: optimizes · Numeric and categorical columns are reduced for memory efficiency
- Ingestion → Long panel: reshapes · Wide retail tables are converted into a long, time-indexed sales panel
- Downcasting → Long panel: produces · Compact panel representation ready for feature generation
- Long panel → Feature build: inputs · The tidy sales panel is the base table for supervised learning
- Feature build → Lag features: adds · Historical demand lags are appended to each training row
- Feature build → Rolling stats: adds · Rolling window summary statistics are computed from recent demand
- Feature build → Calendar covariates: adds · Calendar, event, and price signals are merged in
- Feature build → Popularity encodings: adds · Series and item popularity information is encoded
- Lag features → 28-day holdout: enters · Engineered lag signals are evaluated under temporal holdout
- Rolling stats → 28-day holdout: enters · Rolling features are validated on a future window
- Calendar covariates → 28-day holdout: enters · Exogenous features participate in the split
- Popularity encodings → 28-day holdout: enters · Cross-series encodings are checked against the holdout
- 28-day holdout → LightGBM model: trains · A global LightGBM learner is fit on the training horizon
- LightGBM model → Model artifact: serializes · The trained model is persisted as a reusable artifact
- Model artifact → Streamlit app: loads · Streamlit runtime loads the serialized model for inference
- Streamlit app → Inventory policy: simulates · Forecasts are translated into inventory recommendations and cost tradeoffs
- LightGBM model → Inventory policy: scores · Predictions feed RMSE and asymmetric inventory-loss evaluation
- Forecast notebook → Ingestion: orchestrates · Notebook calls the preprocessing module during the end-to-end pipeline
- Forecast notebook → Feature build: builds · Notebook drives feature matrix construction
- Forecast notebook → LightGBM model: trains · Notebook runs model fitting and persists the result
- Forecast notebook → Inventory policy: evaluates · Notebook computes validation metrics and business-cost checks
- Demand notebook → Feature build: explores · Secondary notebook supports demand forecasting experimentation
