# servicenow/n-beats architecture

> Architecture diagram of the GitHub repository servicenow/n-beats, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/servicenow/n-beats
- Repository: https://github.com/servicenow/n-beats

## Overview

##### Project Type and Purpose
- This is an ML research‐pipeline repo that implements and reproduces the N‑BEATS univariate time‑series forecasting algorithm in PyTorch.
- It provides end-to-end components: dataset download & loading, model definition, experiment orchestration via configurable scripts, result storage, summary scripts, and analysis notebooks.

##### High-Level Components
- CLI/Build Layer
– Makefile targets (init, dataset, build, run, notebook) drive Docker image build, data download, experiment creation, execution, and analysis.
– Dockerfile wraps all dependencies into a container.
- Configuration Layer
– Gin configuration files under experiments/* define hyperparameters and pipeline settings (generic vs. interpretable model).
- Experiment Orchestrator
– `experiments/main.py` and `experiments/trainer.py` read gin configs, instantiate data loaders, models, and training loops.
- Common Utilities
– `common/` holds shared code: experiment scaffolding, HTTP utilities, metrics, samplers, settings, and PyTorch extensions (losses, ops, snapshot).
- Data Layer
– `datasets/` modules implement dataset download and PyTorch-compatible loaders for electricity, M3, M4, tourism, traffic.
- Model Layer
– `models/nbeats.py` defines the neural architecture (stacked fully connected blocks).
- Storage Layer
– Local FS directories under `storage/` (created by make commands) store:
· raw datasets
· experiment configurations and generated commands
· training logs, model snapshots, loss curves, and forecast outputs
- Analysis Layer
– `summary/` scripts compute aggregated metrics from saved forecasts.

##### Component Interactions and Data Flow
1. User invokes Makefile targets → Docker container initialization
2. Makefile → downloads datasets into `storage/datasets` via `datasets/main.py`
3. Makefile build target + gin config → for each ensemble member, writes commands into `storage/experiments/...`
4. Makefile run target → executes each command:
– Trainer script reads gin files, pulls data from `datasets/`, uses common utilities, loads `models/nbeats.py`, trains on GPU/CPU
– Outputs saved snapshots, logs, forecasts into `storage/experiments/...`
5. Summary scripts ingest forecasts, compute metrics, write to `storage/summary`
6. Notebooks connect to `storage/experiments` and `storage/summary` for visualizations
7. (Optional) Cloud scheduler can replace Makefile run loop, submitting each command to a cluster

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI/Build Layer
    subgraph "CLI/Build Layer"
        Make["Makefile"]:::orchestration
        Dockerfile["Dockerfile"]:::orchestration
    end
    Make -->|"build"| Dockerfile

    %% Configuration Layer
    subgraph "Configuration (Gin configs)"
        gen_elec["electricity/generic.gin"]:::config
        intp_elec["electricity/interpretable.gin"]:::config
        gen_m3["m3/generic.gin"]:::config
        intp_m3["m3/interpretable.gin"]:::config
        gen_m4["m4/generic.gin"]:::config
        intp_m4["m4/interpretable.gin"]:::config
        gen_tour["tourism/generic.gin"]:::config
        intp_tour["tourism/interpretable.gin"]:::config
        gen_traffic["traffic/generic.gin"]:::config
        intp_traffic["traffic/interpretable.gin"]:::config
    end
    Dockerfile -->|"loads .gin configs"| gen_elec
    Dockerfile --> intp_elec
    Dockerfile --> gen_m3
    Dockerfile --> intp_m3
    Dockerfile --> gen_m4
    Dockerfile --> intp_m4
    Dockerfile --> gen_tour
    Dockerfile --> intp_tour
    Dockerfile --> gen_traffic
    Dockerfile --> intp_traffic

    %% Experiment Orchestrator
    subgraph "Experiment Orchestrator"
        trainer["experiments/trainer.py"]:::orchestration
        model_py["experiments/model.py"]:::orchestration
        elec_main["experiments/electricity/main.py"]:::orchestration
        m3_main["experiments/m3/main.py"]:::orchestration
        m4_main["experiments/m4/main.py"]:::orchestration
        tour_main["experiments/tourism/main.py"]:::orchestration
        traf_main["experiments/traffic/main.py"]:::orchestration
    end
    gen_elec -->|"configure"| elec_main
    intp_elec --> elec_main
    gen_m3 --> m3_main
    intp_m3 --> m3_main
    gen_m4 --> m4_main
    intp_m4 --> m4_main
    gen_tour --> tour_main
    intp_tour --> tour_main
    gen_traffic --> traf_main
    intp_traffic --> traf_main
    elec_main --> trainer
    m3_main --> trainer
    m4_main --> trainer
    tour_main --> trainer
    traf_main --> trainer
    trainer --> model_py

    %% Codebase Layer
    subgraph "Codebase"
        subgraph "Data Layer"
            datasets_main["datasets/main.py"]:::compute
            elec_ds["datasets/electricity.py"]:::compute
            m3_ds["datasets/m3.py"]:::compute
            m4_ds["datasets/m4.py"]:::compute
            tour_ds["datasets/tourism.py"]:::compute
            traf_ds["datasets/traffic.py"]:::compute
        end
        subgraph "Model Layer"
            nbeats["models/nbeats.py"]:::compute
        end
        subgraph "Common Utilities"
            exp_utils["common/experiment.py"]:::compute
            http_utils["common/http_utils.py"]:::compute
            metrics["common/metrics.py"]:::compute
            sampler["common/sampler.py"]:::compute
            settings["common/settings.py"]:::compute
            losses["common/torch/losses.py"]:::compute
            ops["common/torch/ops.py"]:::compute
            snaps["common/torch/snapshots.py"]:::compute
        end
    end
    trainer --> datasets_main
    datasets_main --> elec_ds
    datasets_main --> m3_ds
    datasets_main --> m4_ds
    datasets_main --> tour_ds
    datasets_main --> traf_ds
    trainer --> exp_utils
    trainer --> metrics
    trainer --> sampler
    trainer --> settings
    trainer --> losses
    trainer --> ops
    trainer --> snaps
    model_py --> nbeats

    %% Storage Layer
    storage["Local FS Storage"]:::storage
    trainer -->|"saves .pt snapshots, logs, forecasts"| storage
    datasets_main -->|"stores raw data"| storage

    %% Analysis Layer
    subgraph "Analysis Layer"
        subgraph "Summary Scripts"
            sum_elec["summary/electricity.py"]:::analysis
            sum_m3["summary/m3.py"]:::analysis
            sum_m4["summary/m4.py"]:::analysis
            sum_tour["summary/tourism.py"]:::analysis
            sum_traf["summary/traffic.py"]:::analysis
            sum_utils["summary/utils.py"]:::analysis
        end
        subgraph "Notebooks"
            nb_elec["notebooks/Electricity.ipynb"]:::analysis
            nb_m3["notebooks/M3.ipynb"]:::analysis
            nb_m4["notebooks/M4.ipynb"]:::analysis
            nb_tour["notebooks/Tourism.ipynb"]:::analysis
            nb_traf["notebooks/Traffic.ipynb"]:::analysis
        end
    end
    storage -->|"reads forecasts"| sum_elec
    storage --> sum_m3
    storage --> sum_m4
    storage --> sum_tour
    storage --> sum_traf
    storage -->|"reads outputs"| nb_elec
    storage --> nb_m3
    storage --> nb_m4
    storage --> nb_tour
    storage --> nb_traf

    %% Optional Cloud Scheduler
    cloud["Cloud Scheduler"]:::external
    compute_nodes["Compute Nodes (GPU/CPU)"]:::external
    trainer -->|"submit jobs"| cloud
    cloud --> compute_nodes
    trainer -->|"direct compute"| compute_nodes

    %% Click Events
    click Make "https://github.com/servicenow/n-beats/tree/master/Makefile"
    click Dockerfile "https://github.com/servicenow/n-beats/tree/master/./Dockerfile"
    click gen_elec "https://github.com/servicenow/n-beats/blob/master/./experiments/electricity/generic.gin"
    click intp_elec "https://github.com/servicenow/n-beats/blob/master/./experiments/electricity/interpretable.gin"
    click gen_m3 "https://github.com/servicenow/n-beats/blob/master/./experiments/m3/generic.gin"
    click intp_m3 "https://github.com/servicenow/n-beats/blob/master/./experiments/m3/interpretable.gin"
    click gen_m4 "https://github.com/servicenow/n-beats/blob/master/./experiments/m4/generic.gin"
    click intp_m4 "https://github.com/servicenow/n-beats/blob/master/./experiments/m4/interpretable.gin"
    click gen_tour "https://github.com/servicenow/n-beats/blob/master/./experiments/tourism/generic.gin"
    click intp_tour "https://github.com/servicenow/n-beats/blob/master/./experiments/tourism/interpretable.gin"
    click gen_traffic "https://github.com/servicenow/n-beats/blob/master/./experiments/traffic/generic.gin"
    click intp_traffic "https://github.com/servicenow/n-beats/blob/master/./experiments/traffic/interpretable.gin"
    click trainer "https://github.com/servicenow/n-beats/blob/master/./experiments/trainer.py"
    click model_py "https://github.com/servicenow/n-beats/blob/master/./experiments/model.py"
    click elec_main "https://github.com/servicenow/n-beats/blob/master/./experiments/electricity/main.py"
    click m3_main "https://github.com/servicenow/n-beats/blob/master/./experiments/m3/main.py"
    click m4_main "https://github.com/servicenow/n-beats/blob/master/./experiments/m4/main.py"
    click tour_main "https://github.com/servicenow/n-beats/blob/master/./experiments/tourism/main.py"
    click traf_main "https://github.com/servicenow/n-beats/blob/master/./experiments/traffic/main.py"
    click exp_utils "https://github.com/servicenow/n-beats/blob/master/./common/experiment.py"
    click http_utils "https://github.com/servicenow/n-beats/blob/master/./common/http_utils.py"
    click metrics "https://github.com/servicenow/n-beats/blob/master/./common/metrics.py"
    click sampler "https://github.com/servicenow/n-beats/blob/master/./common/sampler.py"
    click settings "https://github.com/servicenow/n-beats/blob/master/./common/settings.py"
    click losses "https://github.com/servicenow/n-beats/blob/master/./common/torch/losses.py"
    click ops "https://github.com/servicenow/n-beats/blob/master/./common/torch/ops.py"
    click snaps "https://github.com/servicenow/n-beats/blob/master/./common/torch/snapshots.py"
    click datasets_main "https://github.com/servicenow/n-beats/blob/master/./datasets/main.py"
    click elec_ds "https://github.com/servicenow/n-beats/blob/master/./datasets/electricity.py"
    click m3_ds "https://github.com/servicenow/n-beats/blob/master/./datasets/m3.py"
    click m4_ds "https://github.com/servicenow/n-beats/blob/master/./datasets/m4.py"
    click tour_ds "https://github.com/servicenow/n-beats/blob/master/./datasets/tourism.py"
    click traf_ds "https://github.com/servicenow/n-beats/blob/master/./datasets/traffic.py"
    click nbeats "https://github.com/servicenow/n-beats/blob/master/./models/nbeats.py"
    click sum_elec "https://github.com/servicenow/n-beats/blob/master/./summary/electricity.py"
    click sum_m3 "https://github.com/servicenow/n-beats/blob/master/./summary/m3.py"
    click sum_m4 "https://github.com/servicenow/n-beats/blob/master/./summary/m4.py"
    click sum_tour "https://github.com/servicenow/n-beats/blob/master/./summary/tourism.py"
    click sum_traf "https://github.com/servicenow/n-beats/blob/master/./summary/traffic.py"
    click sum_utils "https://github.com/servicenow/n-beats/blob/master/./summary/utils.py"
    click nb_elec "https://github.com/servicenow/n-beats/blob/master/./notebooks/Electricity.ipynb"
    click nb_m3 "https://github.com/servicenow/n-beats/blob/master/./notebooks/M3.ipynb"
    click nb_m4 "https://github.com/servicenow/n-beats/blob/master/./notebooks/M4.ipynb"
    click nb_tour "https://github.com/servicenow/n-beats/blob/master/./notebooks/Tourism.ipynb"
    click nb_traf "https://github.com/servicenow/n-beats/blob/master/./notebooks/Traffic.ipynb"

    %% Styles
    classDef orchestration fill:#f9c784,stroke:#e69f3e,color:#000
    classDef config fill:#e0bbf1,stroke:#9b59b6,color:#000
    classDef compute fill:#aed6f1,stroke:#2980b9,color:#000
    classDef storage fill:#a3e4d7,stroke:#16a085,color:#000
    classDef analysis fill:#f5b7b1,stroke:#c0392b,color:#000
    classDef external fill:#d5d8dc,stroke:#7f8c8d,color:#000
```
