# aneeshkm/optimizing-temporal-fusion-transformer architecture

> Architecture diagram of the GitHub repository aneeshkm/optimizing-temporal-fusion-transformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aneeshkm/optimizing-temporal-fusion-transformer
- Repository: https://github.com/aneeshkm/optimizing-temporal-fusion-transformer

## Overview

##### Project Type & Purpose
- This is an experimental ML optimization pipeline for the Temporal Fusion Transformer (TFT).
- It integrates baseline training, FlashAttention acceleration, Optuna-based hyperparameter tuning, and Post-Training Quantization (PTQ) to reduce latency, memory, and model size.

##### File Structure Analysis
- Top-level directories represent sequential pipeline stages:
- “Baseline Model Codes” – original TFT experiments (Jupyter notebooks).
- “Flash Attention Codes” – TFT experiments with FlashAttention integration, including Optuna variants.
- “FlashAttention Library Modifications” – a local fork/patch of GluonTS to inject custom FlashAttention kernels.
- “Quantization” – notebooks for PTQ experiments.
- Key config/build artifacts: no monolithic server or web frontend—pure compute notebooks plus Python modules under “local/gluonts”.

##### README Insights
- Describes four-stage experimental pipeline: Baseline → FlashAttention → Hyperparameter Tuning → Quantization.
- Tech stack: PyTorch, GluonTS, FlashAttention, Optuna, PyTorch Quantization on NVIDIA A100/V100 HPC.

##### Main Components
a. Data Ingestion & Preprocessing
- Input time series datasets
- Preprocessing module (GluonTS data pipeline)
b. Baseline Training Module
- Jupyter notebooks invoking GluonTS’ TFT estimator
c. FlashAttention Acceleration Layer
- Patched GluonTS modules under `/local/gluonts/torch/model/tft`
- Custom attention kernels from FlashAttention library
d. Hyperparameter Tuning (Optuna)
- Optuna study objects orchestrating multiple training calls
- Parameter server or local SQLite storage for trials
e. Post-Training Quantization (PTQ)
- PyTorch quantization scripts/notebooks
- Model size compression component
f. Evaluation & Metrics
- GluonTS evaluation and backtest modules under `/local/gluonts/evaluation`
- RMSE, MASE, latency, memory profiling
g. Compute Resources
- NVIDIA A100/V100 GPUs on an HPC cluster
- Job scheduler (e.g., SLURM) if used

##### Relationships & Interactions
- Data → Preprocessing → Baseline Trainer → Evaluation → Metrics Store
- Baseline Trainer → (branch) FlashAttention Trainer → Evaluation
- FlashAttention Trainer + Baseline Trainer feeds Optuna for tuning loops
- Best-performing models → PTQ Module → Quantized Model → Final Evaluation
- All training/evaluation jobs run on GPUs via the HPC scheduler

##### Architectural Patterns & Principles
- Pipeline orchestration: modular, sequential ML stages
- Code injection/monkey-patch: local GluonTS fork for FlashAttention
- Experiment tracking: notebooks + Optuna storage
- Separation of concerns between training, tuning, quantization, and evaluation

##### Final Touches
- Components are fully labeled (e.g., “TFT Baseline Trainer”, “Optuna Study Manager”, “PTQ Quantizer”).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Pipeline Stages
    subgraph "Data Ingestion & Preprocessing"
        Data["Data Ingestion & Preprocessing"]:::code
    end
    click Data "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/FlashAttention Library Modifications/local/gluonts/model/forecast.py"

    subgraph "Baseline Training Module"
        B_nb1["tft-baseline-execution.ipynb"]:::code
        B_nb2["optuna-tft-baseline-execution.ipynb"]:::code
        B_nb3["tft-baseline-execution-best.ipynb"]:::code
    end
    click B_nb1 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/Baseline Model Codes/Baseline Code/tft-baseline-execution.ipynb"
    click B_nb2 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/Baseline Model Codes/Baseline + Optuna/optuna-tft-baseline-execution.ipynb"
    click B_nb3 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/Baseline Model Codes/Baseline Best/tft-baseline-execution-best.ipynb"

    subgraph "FlashAttention Acceleration Layer"
        FA_mod1["estimator.py"]:::code
        FA_mod2["layers.py"]:::code
        FA_mod3["lightning_module.py"]:::code
        FA_nb1["tft-baseline-execution-1-epoch.ipynb"]:::code
        FA_nb2["tft-flashAttention-execution-best.ipynb"]:::code
    end
    click FA_mod1 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/FlashAttention Library Modifications/local/gluonts/torch/model/tft/estimator.py"
    click FA_mod2 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/FlashAttention Library Modifications/local/gluonts/torch/model/tft/layers.py"
    click FA_mod3 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/FlashAttention Library Modifications/local/gluonts/torch/model/tft/lightning_module.py"
    click FA_nb1 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/Flash Attention Codes/Baseline Code + Flash attention/tft-baseline-execution-1-epoch.ipynb"
    click FA_nb2 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/Flash Attention Codes/Flash Attention Best/tft-flashAttention-execution-best.ipynb"

    subgraph "Hyperparameter Tuning (Optuna)"
        Opt_nb1["optuna-tft-flashAttention-execution.ipynb"]:::code
        Opt_nb2["optuna-tft-baseline-execution.ipynb"]:::code
        Opt_DB["Optuna SQLite DB"]:::storage
    end
    click Opt_nb1 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/Flash Attention Codes/Flash Attention Optuna/optuna-tft-flashAttention-execution.ipynb"
    click Opt_nb2 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/Baseline Model Codes/Baseline + Optuna/optuna-tft-baseline-execution.ipynb"

    subgraph "Post-Training Quantization (PTQ)"
        PTQ_nb1["Post-Training Quantization.ipynb"]:::code
        PTQ_nb2["PTQ-Large-Parameters.ipynb"]:::code
    end
    click PTQ_nb1 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/Quantization/Post-Training Quantization.ipynb"
    click PTQ_nb2 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/Quantization/PTQ-Large-Parameters.ipynb"

    subgraph "Evaluation & Metrics"
        Eval_mod1["_base.py"]:::code
        Eval_mod2["backtest.py"]:::code
        Eval_mod3["metrics.py"]:::code
        Metrics_DB["Metrics Store"]:::storage
    end
    click Eval_mod1 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/FlashAttention Library Modifications/local/gluonts/evaluation/_base.py"
    click Eval_mod2 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/FlashAttention Library Modifications/local/gluonts/evaluation/backtest.py"
    click Eval_mod3 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/FlashAttention Library Modifications/local/gluonts/evaluation/metrics.py"

    subgraph "Local GluonTS Fork"
        GF1["__init__.py"]:::code
        GF2["forecast.py"]:::code
        GF3["module.py"]:::code
    end
    click GF1 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/FlashAttention Library Modifications/local/gluonts/__init__.py"
    click GF2 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/FlashAttention Library Modifications/local/gluonts/model/forecast.py"
    click GF3 "https://github.com/aneeshkm/optimizing-temporal-fusion-transformer/blob/main/FlashAttention Library Modifications/local/gluonts/torch/model/tft/module.py"

    %% Compute Resources
    Cluster("HPC GPU Cluster"):::compute

    %% Flow
    Data -->|".ipynb, Python import"| B_nb1
    Data -->|".ipynb, Python import"| FA_mod1
    B_nb1 -->|".model"| Eval_mod1
    FA_mod1 -->|".model"| Eval_mod1
    B_nb1 -->|trials| Opt_DB
    FA_mod1 -->|trials| Opt_DB
    Opt_DB -->|params| Opt_nb1
    Opt_nb1 -->|best_model| PTQ_nb1
    PTQ_nb1 -->|quantized_model| Eval_mod1
    Eval_mod3 -->|metrics| Metrics_DB

    %% Compute scheduling
    Cluster --> B_nb1
    Cluster --> FA_nb1
    Cluster --> Opt_nb1
    Cluster --> PTQ_nb1
    Cluster --> Eval_mod1

    %% Styles
    classDef code fill:#D0E6FF,stroke:#0366D6,color:#0366D6,stroke-width:1px;
    classDef storage fill:#FFF5BA,stroke:#D69C00,color:#8A5800,stroke-width:1px,stroke-dasharray: 5 5;
    classDef compute fill:#FFE6D0,stroke:#D96400,color:#A63D00,stroke-width:1px;
    classDef external fill:#E8FFED,stroke:#28A745,color:#1C531C,stroke-width:1px;
```
