# imhgchoi/arima-lstm-hybrid-corrcoef-predict architecture

> Architecture diagram of the GitHub repository imhgchoi/arima-lstm-hybrid-corrcoef-predict, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict
- Repository: https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict

## Overview

##### Project Type and Purpose
- This is a research‐oriented data science pipeline for predicting future price correlation coefficients between two assets using an ARIMA‐LSTM hybrid model.
- It is organized around Jupyter notebooks for data scraping, preprocessing, ARIMA modeling, LSTM modeling, and evaluation, supported by raw Python scripts and saved model artifacts.

##### File Structure
- Top‐level notebooks:
- Data_Scraping_Codes.ipynb (web scraping)
- DATA_PREPROCESSING.ipynb (cleaning & reshaping)
- ARIMA_MODELING.ipynb & NEW_ASSET_ARIMA_MODELING.ipynb (ARIMA training & inference)
- MODEL_EVALUATOR.ipynb (benchmarking vs. other financial models)
- MISCELLANEOUS.ipynb
- Supporting directories:
- stock_data/ (raw CSVs per ticker)
- train_dev_test/ (split datasets before/after ARIMA)
- raw_python_codes/ (scripts for LSTM training & new asset testing)
- models/hybrid_LSTM/ (Keras .h5 checkpoints & metrics CSV)
- Key data files: SP500_list.csv, SP500_index.csv, stock08_price.csv, SP500_index.csv
- Config & dependencies implied by README: BeautifulSoup, Quandl, pyramid‐arima, Keras.

##### README Insights
- The README outlines three logical parts:
1. Dataset preparation
2. ARIMA modeling & residual generation
3. LSTM modeling & evaluation
- It specifies end‐user steps (data fetch → windowed correlation → ARIMA fit & residual forecast → LSTM inference → combine results).

##### Main Components
a. Data Ingestion Layer
- Web Scraper (BeautifulSoup) → SP500_list.csv
- Financial API Client (Quandl) → stock_data/*.csv + SP500_index.csv
b. Data Storage
- Raw CSV store (stock_data/, SP500_*.csv)
- Processed dataset (stock08_price.csv)
- Train/Dev/Test splits (train_dev_test/)
- Model artifacts (models/hybrid_LSTM/)
c. Data Processing Layer
- DATA_PREPROCESSING notebook: NA imputation, time‐window correlation (train_dev_test/before_arima)
d. ARIMA Modeling Module
- ARIMA_MODELING notebook: fits ARIMA, outputs residual series & forecast (train_dev_test/after_arima)
- NEW_ASSET_ARIMA_MODELING notebook for bulk/incremental ARIMA inference
e. LSTM Modeling Module
- raw_python_codes/Residual_LSTM.py: defines LSTM architecture
- raw_python_codes/new_asset_testing_*.py: inference pipelines
- models/hybrid_LSTM/: saved checkpoints per epoch, metrics CSV
f. Model Evaluation & Benchmarking
- MODEL_EVALUATOR notebook: compares hybrid model against baselines
- MISCELLANEOUS for ad‐hoc experiments
g. Final Prediction Flow
- Combine ARIMA forecast + LSTM residual prediction → final correlation prediction

##### Relationships & Data Flow
1. Data Sources → Data Preprocessing
2. Preprocessed Data → ARIMA Module → residuals + one‐step forecast
3. Residuals → LSTM Module → residual forecast
4. ARIMA forecast + LSTM output → Combine → Final result
5. Final result → Model Evaluation
6. Artifacts stored back into models/ and train_dev_test/
- Distinct pipelines for training versus inference (new_asset scripts).

##### Architectural Patterns & Design Principles
- Pipeline pattern: sequential transformation stages
- Modular design: each notebook/script handles a single concern
- Separation of training vs. inference logic
- Reusable components (raw_python_codes) for LSTM deployment
- Clear data‐at‐rest (CSV directories) vs. data‐in‐motion (notebook cells/scripts)

##### Relevant Technologies, Frameworks & Libraries
- BeautifulSoup (scraping)
- Quandl client (data fetch)
- pandas/numpy (data manipulation)
- pyramid‐arima (ARIMA fit & forecast)
- Keras (LSTM model)
- Jupyter notebooks (orchestration & documentation)

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Services
    subgraph "Data Ingestion Layer"
        direction TB
        WS["Data_Scraping_Codes.ipynb"]:::processing
        QA["Quandl Client"]:::external
        WS -->|"SP500_list.csv"| SP500_LIST["SP500_list.csv"]:::datastore
        WS -->|"SP500_index.csv"| SP500_INDEX["SP500_index.csv"]:::datastore
        QA -->|"stock_data/*.csv"| STOCK_DATA["stock_data/"]:::datastore
    end

    %% Data Storage
    subgraph "Data Storage" 
        direction TB
        STOCK08["stock08_price.csv"]:::datastore
        BEFORE_ARIMA["train_dev_test/before_arima/"]:::datastore
        AFTER_ARIMA["train_dev_test/after_arima/"]:::datastore
        NEW_ASSET_BEFORE["new_asset_before_arima.csv"]:::datastore
        NEW_ASSET_AFTER["new_asset_after_arima.csv"]:::datastore
        LSTM_STORE["models/hybrid_LSTM/"]:::datastore
        LSTM_METRICS["models/hybrid_LSTM.csv"]:::datastore
    end

    %% Preprocessing
    subgraph "Data Processing Layer"
        direction TB
        DP["DATA_PREPROCESSING.ipynb"]:::processing
    end

    %% ARIMA Modeling
    subgraph "ARIMA Modeling Module"
        direction TB
        ARIMA_NB["ARIMA_MODELING.ipynb"]:::modeling
        NEW_ARIMA_NB["NEW_ASSET_ARIMA_MODELING.ipynb"]:::modeling
    end

    %% LSTM Modeling
    subgraph "LSTM Modeling Module"
        direction TB
        LSTM_SCRIPT["Residual_LSTM.py"]:::modeling
        NEW_ASSET_BEFORE_SCRIPT["new_asset_testing_beforeARIMA.py"]:::modeling
        NEW_ASSET_AFTER_SCRIPT["new_asset_testing_afterARIMA.py"]:::modeling
    end

    %% Evaluation
    subgraph "Model Evaluation & Benchmarking"
        direction TB
        EVAL["MODEL_EVALUATOR.ipynb"]:::processing
        MISC["MISCELLANEOUS.ipynb"]:::processing
    end

    %% Config & Assets
    subgraph "Documentation & Assets"
        direction TB
        README["README.md"]:::external
        IMAGES["images/"]:::datastore
    end

    %% Data Flow
    SP500_LIST -->|raw data| STOCK08
    SP500_INDEX -->|raw data| STOCK08
    STOCK_DATA -->|raw data| STOCK08

    STOCK08 -->|processed data| DP
    DP -->|train/dev/test splits| BEFORE_ARIMA

    BEFORE_ARIMA -->|train_X/Y| ARIMA_NB
    ARIMA_NB -->|residuals & forecast| AFTER_ARIMA

    AFTER_ARIMA -->|residuals| LSTM_SCRIPT
    NEW_ASSET_BEFORE -->|new asset splits| NEW_ARIMA_NB
    NEW_ARIMA_NB -->|new residuals| NEW_ASSET_AFTER

    LSTM_SCRIPT -->|train LSTM| LSTM_STORE
    LSTM_STORE -->|metrics| LSTM_METRICS

    AFTER_ARIMA -->|inference residuals| NEW_ASSET_AFTER_SCRIPT
    NEW_ASSET_AFTER_SCRIPT -->|predicted residuals| NEW_ASSET_AFTER

    ARIMA_NB -->|forecast| COMBINE[/"Combine ARIMA + LSTM"/]:::processing
    LSTM_SCRIPT -->|residual forecast| COMBINE
    COMBINE -->|final prediction| EVAL
    COMBINE -->|final prediction| MISC

    %% Click Events
    click WS "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/Data_Scraping_Codes.ipynb"
    click SP500_LIST "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/SP500_list.csv"
    click SP500_INDEX "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/SP500_index.csv"
    click STOCK_DATA "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/tree/master/stock_data/"
    click STOCK08 "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/stock08_price.csv"
    click DP "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/DATA_PREPROCESSING.ipynb"
    click BEFORE_ARIMA "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/tree/master/train_dev_test/before_arima/"
    click ARIMA_NB "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/ARIMA_MODELING.ipynb"
    click NEW_ARIMA_NB "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/NEW_ASSET_ARIMA_MODELING.ipynb"
    click AFTER_ARIMA "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/tree/master/train_dev_test/after_arima/"
    click NEW_ASSET_BEFORE "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/train_dev_test/new_asset_before_arima.csv"
    click NEW_ASSET_AFTER "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/train_dev_test/new_asset_after_arima.csv"
    click LSTM_SCRIPT "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/raw_python_codes/Residual_LSTM.py"
    click NEW_ASSET_BEFORE_SCRIPT "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/raw_python_codes/new_asset_testing_beforeARIMA.py"
    click NEW_ASSET_AFTER_SCRIPT "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/raw_python_codes/new_asset_testing_afterARIMA.py"
    click LSTM_STORE "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/tree/master/models/hybrid_LSTM/"
    click LSTM_METRICS "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/models/hybrid_LSTM.csv"
    click EVAL "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/MODEL_EVALUATOR.ipynb"
    click MISC "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/MISCELLANEOUS.ipynb"
    click README "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/blob/master/README.md"
    click IMAGES "https://github.com/imhgchoi/arima-lstm-hybrid-corrcoef-predict/tree/master/images/"

    %% Styles
    classDef datastore fill:#d3d3d3,stroke:#333,stroke-width:1px
    classDef processing fill:#b2fab4,stroke:#333,stroke-width:1px
    classDef modeling fill:#ffd699,stroke:#333,stroke-width:1px
    classDef external fill:#9fc5e8,stroke:#333,stroke-width:1px
```
