How aneeshkm/optimizing-temporal-fusion-transformer works
Project Type & Purpose
• This is an experimental ML optimization pipeline for the Temporal Fusion Transformer (TFT).
• It integrates baseline training, FlashAttention acceleration, Optuna-based hyperparameter tuning, and Post-Training Quantization (PTQ) to reduce latency, memory, and model size.
File Structure Analysis
• Top-level directories represent sequential pipeline stages:
• “Baseline Model Codes” – original TFT experiments (Jupyter notebooks).
• “Flash Attention Codes” – TFT experiments with FlashAttention integration, including Optuna variants.
• “FlashAttention Library Modifications” – a local fork/patch of GluonTS to inject custom FlashAttention kernels.
• “Quantization” – notebooks for PTQ experiments.
• Key config/build artifacts: no monolithic server or web frontend—pure compute notebooks plus Python modules under “local/gluonts”.
README Insights
• Describes four-stage experimental pipeline: Baseline → FlashAttention → Hyperparameter Tuning → Quantization.
• Tech stack: PyTorch, GluonTS, FlashAttention, Optuna, PyTorch Quantization on NVIDIA A100/V100 HPC.
Main Components
a. Data Ingestion & Preprocessing
• Input time series datasets
• Preprocessing module (GluonTS data pipeline)
b. Baseline Training Module
• Jupyter notebooks invoking GluonTS’ TFT estimator
c. FlashAttention Acceleration Layer
• Patched GluonTS modules under /local/gluonts/torch/model/tft
• Custom attention kernels from FlashAttention library
d. Hyperparameter Tuning (Optuna)
• Optuna study objects orchestrating multiple training calls
• Parameter server or local SQLite storage for trials
e. Post-Training Quantization (PTQ)
• PyTorch quantization scripts/notebooks
• Model size compression component
f. Evaluation & Metrics
• GluonTS evaluation and backtest modules under /local/gluonts/evaluation
• RMSE, MASE, latency, memory profiling
g. Compute Resources
• NVIDIA A100/V100 GPUs on an HPC cluster
• Job scheduler (e.g., SLURM) if used
Relationships & Interactions
• Data → Preprocessing → Baseline Trainer → Evaluation → Metrics Store
• Baseline Trainer → (branch) FlashAttention Trainer → Evaluation
• FlashAttention Trainer + Baseline Trainer feeds Optuna for tuning loops
• Best-performing models → PTQ Module → Quantized Model → Final Evaluation
• All training/evaluation jobs run on GPUs via the HPC scheduler
Architectural Patterns & Principles
• Pipeline orchestration: modular, sequential ML stages
• Code injection/monkey-patch: local GluonTS fork for FlashAttention
• Experiment tracking: notebooks + Optuna storage
• Separation of concerns between training, tuning, quantization, and evaluation
Final Touches
• Components are fully labeled (e.g., “TFT Baseline Trainer”, “Optuna Study Manager”, “PTQ Quantizer”).