# pmoll01/coin2025_vollvertnuetelt architecture

> Architecture diagram of the GitHub repository pmoll01/coin2025_vollvertnuetelt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pmoll01/coin2025_vollvertnuetelt
- Repository: https://github.com/pmoll01/coin2025_vollvertnuetelt

## Overview

##### Project Type & Purpose
- This is a full-stack data science pipeline for **interpretable stock market forecasting**.
- Key features:
- Ingest and preprocess two raw data sources (financial time series and Twitter sentiment).
- Engineer time-series features.
- Train & evaluate a **Temporal Fusion Transformer (TFT)** model.
- Generate forecasts and interpretability plots.

##### High-Level Components
a. Data Layer
- Raw data:
- `data/finance_data/raw…` (CSV, notebooks)
- `data/twitter_data/raw…` (CSV, notebooks)
- Processed data:
- Feature scripts in `data/finance_data/*.py` & `data/twitter_data/*.ipynb`
- Consolidated CSVs under `data/…/processed/`
- Dataset builder: `data/create_dataset.py` & `data/Pipeline.py`

b. Modeling Layer
- Model definitions:
- `model/lstm_*.py`, `model/xgboost_*.py` (baselines)
- `model/tft_model.py` (core TFT class)
- Dependencies: PyTorch Lightning, PyTorch Forecasting

c. Training & Inference
- (Per README—assume)
- `train/train_tft.py`: orchestrates data loading, dataset split, training, checkpointing
- `predict/tft_inference.py`: loads checkpoint, produces next-day forecast
- `predict/interpret.py`: extracts attention weights for feature importances

d. Configuration & Utilities
- `environment.yml` / `requirements.txt` for reproducible env
- `utils/config.py` for paths, batch sizes, learning rates

e. Outputs
- Model checkpoints under `outputs/checkpoints/`
- Plots under `outputs/plots/`

##### Relationships & Data Flow
1. Raw Data → Preprocessing
- Financial & Twitter scripts output cleaned, feature-rich CSVs.
2. Cleaned CSVs → Dataset Creation
- `create_dataset.py` merges features, adds target variable (log return).
3. Dataset → Training
- `train_tft.py` builds a `TimeSeriesDataSet`, trains TFT → checkpoint.
4. Checkpoint → Inference
- `tft_inference.py` loads model, produces prediction.
5. Checkpoint + Dataset → Interpretability
- `interpret.py` visualizes attention weights → feature importance plot.

##### Architectural Patterns & Design Principles
- Modular pipeline: each stage (ingest, preprocess, feature engineer, model, infer, interpret) is isolated.
- Separation of concerns:
- Data folder for ETL notebooks and scripts.
- Model folder for algorithms.
- train/ & predict/ for orchestration (as indicated in README).
- Reproducibility: environment specification, config centralization.
- Extensibility: easy to swap models (LSTM, XGBoost, TFT) or add features.

##### Essential Details to Capture
- All 3 data types: finance, Twitter, and merged dataset.
- The **pipeline orchestrator** (`Pipeline.py`).
- The **TFT model** and its checkpoints.
- The **interpretability step** extracting attention weights.
- Configuration as a central node for parameters.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        direction TB
        FinanceRaw[(“Finance CSV Raw”)]:::data
        TwitterRaw[(“Twitter CSV Raw”)]:::data
        A_finance_notebook{"A_finance_data_scrape.ipynb"}:::data
        B_finance_notebook{"B_finance_data_process.ipynb"}:::data
        FinanceRaw --> A_finance_notebook
        FinanceRaw --> B_finance_notebook
        TwitterRaw --> TwitterRawInspect{"0_Data_inspection.ipynb"}:::data
        TwitterRawInspect --> TwitterPreproc{"1. Data_preprocessing.ipynb"}:::data
    end

    %% Feature Engineering Layer
    subgraph "Feature Engineering Layer"
        direction TB
        run_finance_proc["run_all_feature_processing.py"]:::processing
        subgraph "Finance Feature Scripts"
            direction TB
            f0["0_target_shift_processing.py"]:::processing
            f1["1_feature_moving_averages_processing.py"]:::processing
            f2["2_feature_macd_processing.py"]:::processing
            f3["3_feature_rsi_processing.py"]:::processing
            f4["4_feature_bollinger_bands_processing.py"]:::processing
            f5["5_feature_atr_processing.py"]:::processing
            f6["6_feature_adx_processing.py"]:::processing
            f7["7_feature_stochastic_oscillator_processing.py"]:::processing
            f8["8_feature_obv_processing.py"]:::processing
            f9["9_feature_momentum_roc_processing.py"]:::processing
            f11["11_feature_mfi_processing.py"]:::processing
            f12["12_feature_volatility_volume_processing.py"]:::processing
            f13["13_feature_cross_asset_indicators_processing.py"]:::processing
            f14["14_drop_high_low_features.py"]:::processing
        end
        subgraph "Twitter Feature Notebooks"
            direction TB
            TwitterFeat{"2. Feature_engineering.ipynb"}:::processing
            TwitterPost{"2.1 Feature_postprocessing.ipynb"}:::processing
            TwitterViz{"3. Feature_visualizations.ipynb"}:::processing
            TwitterPhase{"4. Phase_analysis.ipynb"}:::processing
        end
        run_finance_proc --> f0 & f1 & f2 & f3 & f4 & f5 & f6 & f7 & f8 & f9 & f11 & f12 & f13 & f14
        TwitterPreproc --> TwitterFeat --> TwitterPost --> TwitterViz --> TwitterPhase
        f0 & TwitterPhase --> CleanFinanceCSV[(“Finance Features CSV”)]:::data
        TwitterPhase --> CleanTwitterCSV[(“Twitter Features CSV”)]:::data
    end

    %% Dataset Builder Layer
    subgraph "Dataset Builder Layer"
        direction TB
        create_dataset["create_dataset.py"]:::processing
        pipeline_orch["Pipeline.py"]:::processing
        CleanFinanceCSV --> create_dataset
        CleanTwitterCSV --> create_dataset
        create_dataset --> MergedDataset[(“Merged Dataset CSV”)]:::data
    end

    %% Modeling Layer
    subgraph "Modeling Layer"
        direction TB
        subgraph "Baseline Models"
            direction TB
            LSTMcls["lstm_classifier.py"]:::modeling
            LSTMreg1["lstm_regressor_gemini.py"]:::modeling
            LSTMreg2["lstm_regressor_new.py"]:::modeling
            XGBcls["xgboost_classifier.py"]:::modeling
            XGBreg["xgboost_regressor.py"]:::modeling
        end
        TFT["tft_model.py"]:::modeling
        MergedDataset --> LSTMcls & LSTMreg1 & LSTMreg2 & XGBcls & XGBreg & TFT
    end

    %% Training & Inference Layer
    subgraph "Training & Inference Layer"
        direction TB
        Train["train_tft.py"]:::processing
        Inference["tft_inference.py"]:::processing
        Interpret["interpret.py"]:::processing
        MergedDataset --> Train
        Train --> Checkpoint[(“TFT Checkpoint”)]:::output
        Checkpoint --> Inference
        Checkpoint --> Interpret
        MergedDataset --> Interpret
        Inference --> Forecasts[(“Forecast Outputs”)]:::output
        Interpret --> Plots[(“Interpretability Plots”)]:::output
    end

    %% Config & Environment
    Config["config.py"]:::processing
    EnvYml["environment.yml"]:::processing
    ReqTxt["requirements.txt"]:::processing
    Config --> run_finance_proc & TwitterPreproc & create_dataset & Train & Inference & Interpret
    EnvYml --> Config
    ReqTxt --> Config

    %% Click Events
    click run_finance_proc "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/run_all_feature_processing.py"
    click f0 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/0_target_shift_processing.py"
    click f1 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/1_feature_moving_averages_processing.py"
    click f2 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/2_feature_macd_processing.py"
    click f3 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/3_feature_rsi_processing.py"
    click f4 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/4_feature_bollinger_bands_processing.py"
    click f5 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/5_feature_atr_processing.py"
    click f6 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/6_feature_adx_processing.py"
    click f7 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/7_feature_stochastic_oscillator_processing.py"
    click f8 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/8_feature_obv_processing.py"
    click f9 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/9_feature_momentum_roc_processing.py"
    click f11 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/11_feature_mfi_processing.py"
    click f12 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/12_feature_volatility_volume_processing.py"
    click f13 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/13_feature_cross_asset_indicators_processing.py"
    click f14 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/14_drop_high_low_features.py"
    click A_finance_notebook "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/A_finance_data_scrape.ipynb"
    click B_finance_notebook "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/B_finance_data_process.ipynb"
    click run_finance_proc "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/finance_data/run_all_feature_processing.py"
    click TwitterRawInspect "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/twitter_data/0_Data_inspection.ipynb"
    click TwitterPreproc "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/twitter_data/1. Data_preprocessing.ipynb"
    click TwitterFeat "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/twitter_data/2. Feature_engineering.ipynb"
    click TwitterPost "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/twitter_data/2.1 Feature_postprocessing.ipynb"
    click TwitterViz "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/twitter_data/3. Feature_visualizations.ipynb"
    click TwitterPhase "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/twitter_data/4. Phase_analysis.ipynb"
    click create_dataset "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/create_dataset.py"
    click pipeline_orch "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/data/Pipeline.py"
    click LSTMcls "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/model/lstm_classifier.py"
    click LSTMreg1 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/model/lstm_regressor_gemini.py"
    click LSTMreg2 "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/model/lstm_regressor_new.py"
    click XGBcls "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/model/xgboost_classifier.py"
    click XGBreg "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/model/xgboost_regressor.py"
    click TFT "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/model/tft_model.py"
    click EnvYml "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/environment.yml"
    click ReqTxt "https://github.com/pmoll01/coin2025_vollvertnuetelt/blob/main/requirements.txt"

    %% Styles
    classDef data fill:#add8e6,stroke:#000
    classDef processing fill:#ffa500,stroke:#000
    classDef modeling fill:#90ee90,stroke:#000
    classDef output fill:#dda0dd,stroke:#000
```
