How pmoll01/coin2025_vollvertnuetelt works
Project Type & Purpose
• This is a full-stack data science pipeline for interpretable stock market forecasting.
• Key features:
• Ingest and preprocess two raw data sources (financial time series and Twitter sentiment).
• Engineer time-series features.
• Train & evaluate a Temporal Fusion Transformer (TFT) model.
• Generate forecasts and interpretability plots.
High-Level Components
a. Data Layer
• Raw data:
• data/finance_data/raw… (CSV, notebooks)
• data/twitter_data/raw… (CSV, notebooks)
• Processed data:
• Feature scripts in data/finance_data/*.py & data/twitter_data/*.ipynb
• Consolidated CSVs under data/…/processed/
• Dataset builder: data/create_dataset.py & data/Pipeline.py
b. Modeling Layer
• Model definitions:
• model/lstm_*.py, model/xgboost_*.py (baselines)
• model/tft_model.py (core TFT class)
• Dependencies: PyTorch Lightning, PyTorch Forecasting
c. Training & Inference
• (Per README—assume)
• train/train_tft.py: orchestrates data loading, dataset split, training, checkpointing
• predict/tft_inference.py: loads checkpoint, produces next-day forecast
• predict/interpret.py: extracts attention weights for feature importances
d. Configuration & Utilities
• environment.yml / requirements.txt for reproducible env
• utils/config.py for paths, batch sizes, learning rates
e. Outputs
• Model checkpoints under outputs/checkpoints/
• Plots under outputs/plots/
Relationships & Data Flow
1. Raw Data → Preprocessing
• Financial & Twitter scripts output cleaned, feature-rich CSVs.
2. Cleaned CSVs → Dataset Creation
• create_dataset.py merges features, adds target variable (log return).
3. Dataset → Training
• train_tft.py builds a TimeSeriesDataSet, trains TFT → checkpoint.
4. Checkpoint → Inference
• tft_inference.py loads model, produces prediction.
5. Checkpoint + Dataset → Interpretability
• interpret.py visualizes attention weights → feature importance plot.
Architectural Patterns & Design Principles
• Modular pipeline: each stage (ingest, preprocess, feature engineer, model, infer, interpret) is isolated.
• Separation of concerns:
• Data folder for ETL notebooks and scripts.
• Model folder for algorithms.
• train/ & predict/ for orchestration (as indicated in README).
• Reproducibility: environment specification, config centralization.
• Extensibility: easy to swap models (LSTM, XGBoost, TFT) or add features.
Essential Details to Capture
• All 3 data types: finance, Twitter, and merged dataset.
• The pipeline orchestrator (Pipeline.py).
• The TFT model and its checkpoints.
• The interpretability step extracting attention weights.
• Configuration as a central node for parameters.