How imhgchoi/arima-lstm-hybrid-corrcoef-predict works
Project Type and Purpose
• This is a research‐oriented data science pipeline for predicting future price correlation coefficients between two assets using an ARIMA‐LSTM hybrid model.
• It is organized around Jupyter notebooks for data scraping, preprocessing, ARIMA modeling, LSTM modeling, and evaluation, supported by raw Python scripts and saved model artifacts.
File Structure
• Top‐level notebooks:
• Data_Scraping_Codes.ipynb (web scraping)
• DATA_PREPROCESSING.ipynb (cleaning & reshaping)
• ARIMA_MODELING.ipynb & NEW_ASSET_ARIMA_MODELING.ipynb (ARIMA training & inference)
• MODEL_EVALUATOR.ipynb (benchmarking vs. other financial models)
• MISCELLANEOUS.ipynb
• Supporting directories:
• stock_data/ (raw CSVs per ticker)
• train_dev_test/ (split datasets before/after ARIMA)
• raw_python_codes/ (scripts for LSTM training & new asset testing)
• models/hybrid_LSTM/ (Keras .h5 checkpoints & metrics CSV)
• Key data files: SP500_list.csv, SP500_index.csv, stock08_price.csv, SP500_index.csv
• Config & dependencies implied by README: BeautifulSoup, Quandl, pyramid‐arima, Keras.
README Insights
• The README outlines three logical parts:
1. Dataset preparation
2. ARIMA modeling & residual generation
3. LSTM modeling & evaluation
• It specifies end‐user steps (data fetch → windowed correlation → ARIMA fit & residual forecast → LSTM inference → combine results).
Main Components
a. Data Ingestion Layer
• Web Scraper (BeautifulSoup) → SP500_list.csv
• Financial API Client (Quandl) → stock_data/*.csv + SP500_index.csv
b. Data Storage
• Raw CSV store (stock_data/, SP500_*.csv)
• Processed dataset (stock08_price.csv)
• Train/Dev/Test splits (train_dev_test/)
• Model artifacts (models/hybrid_LSTM/)
c. Data Processing Layer
• DATA_PREPROCESSING notebook: NA imputation, time‐window correlation (train_dev_test/before_arima)
d. ARIMA Modeling Module
• ARIMA_MODELING notebook: fits ARIMA, outputs residual series & forecast (train_dev_test/after_arima)
• NEW_ASSET_ARIMA_MODELING notebook for bulk/incremental ARIMA inference
e. LSTM Modeling Module
• raw_python_codes/Residual_LSTM.py: defines LSTM architecture
• raw_python_codes/new_asset_testing_*.py: inference pipelines
• models/hybrid_LSTM/: saved checkpoints per epoch, metrics CSV
f. Model Evaluation & Benchmarking
• MODEL_EVALUATOR notebook: compares hybrid model against baselines
• MISCELLANEOUS for ad‐hoc experiments
g. Final Prediction Flow
• Combine ARIMA forecast + LSTM residual prediction → final correlation prediction
Relationships & Data Flow
1. Data Sources → Data Preprocessing
2. Preprocessed Data → ARIMA Module → residuals + one‐step forecast
3. Residuals → LSTM Module → residual forecast
4. ARIMA forecast + LSTM output → Combine → Final result
5. Final result → Model Evaluation
6. Artifacts stored back into models/ and train_dev_test/
• Distinct pipelines for training versus inference (new_asset scripts).
Architectural Patterns & Design Principles
• Pipeline pattern: sequential transformation stages
• Modular design: each notebook/script handles a single concern
• Separation of training vs. inference logic
• Reusable components (raw_python_codes) for LSTM deployment
• Clear data‐at‐rest (CSV directories) vs. data‐in‐motion (notebook cells/scripts)
Relevant Technologies, Frameworks & Libraries
• BeautifulSoup (scraping)
• Quandl client (data fetch)
• pandas/numpy (data manipulation)
• pyramid‐arima (ARIMA fit & forecast)
• Keras (LSTM model)
• Jupyter notebooks (orchestration & documentation)