# borisbanushev/stockpredictionai architecture

> Architecture diagram of the GitHub repository borisbanushev/stockpredictionai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/borisbanushev/stockpredictionai
- Repository: https://github.com/borisbanushev/stockpredictionai

## Overview

##### Project type and purpose
- This is a research‐oriented data science/ML project implemented as a Jupyter notebook.
- Purpose: ingest multiple sources of time-series and textual data, engineer features (technical indicators, sentiment via BERT, Fourier/ARIMA transforms, autoencoder features, anomaly scores), then train a GAN (LSTM generator + 1D-CNN discriminator) to predict future stock price movements. A reinforcement-learning + Bayesian optimization loop tunes hyperparameters.

##### Main components
- Historical market data (72 correlated assets CSV)
- News feed (daily GS news for BERT-based sentiment)
- Derivatives data (options prices)
- CSV reader, date parser
- Statistical checks (heteroskedasticity, multicollinearity, serial correlation)
1. Technical indicators (MA, EMA, MACD, Bollinger)
2. Fundamental sentiment (BERT/NLP)
3. Fourier transforms & trend extraction
4. ARIMA feature
5. Autoencoder features (stacked VAE + PCA)
6. Anomaly detection (Self-Organizing Maps on options pricing)
7. Feature importance (XGBoost)
1. GAN environment
- Generator: one-layer LSTM (112 input features → 500 hidden → 1 output)
- Discriminator: 1D-CNN (Conv1D → BatchNorm → LeakyReLU → Dense → output)
- Loss: WGAN + Metropolis-Hastings sampling
2. Training loop: alternating D/G updates, learning-rate scheduler (triangular/cyclical)
- RL agent (Rainbow DQN & PPO) interacts with the GAN training environment
- Reward function = 2*G_loss + D_loss + G_accuracy
- Bayesian Optimization (Gaussian process) suggests next hyperparams
- Feedback loop into model training
- Plot real vs. predicted prices after 1, 50, 200 epochs
- Final performance charts

##### Relationships and Interactions
- Data Sources → Data Ingestion
- Data Ingestion → Feature Engineering
- Feature Engineering → Model Training
- Training produces a reward
- Optimizer returns new hyperparameter settings
- Model Training → Evaluation & Visualization

##### Architectural patterns and design principles
- Pipeline/Micro-batch pattern for sequential data processing steps
- Modular separation: data, features, model, tuning, and results
- Feedback loops: hyperparameter tuner → training → reward
- Use of asynchronous or batched updates in GAN training
- Reinforcement Learning as controller for continuous optimization

##### Technologies, frameworks, libraries
- Data processing: pandas, NumPy, statsmodels, scikit-learn
- Feature engineering: fft (NumPy), ARIMA (statsmodels), PCA (scikit-learn), XGBoost
- NLP: BERT (MXNet/Gluon)
- Deep learning: MXNet/Gluon (LSTM, Variational AE, Conv1D)
- GAN improvements: WGAN, Metropolis-Hastings
- Hyperparameter tuning: bayes_opt (BayesianOptimization), RL (OpenAI baselines for PPO, custom Rainbow)
- Visualization: matplotlib, seaborn

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Swimlanes
    subgraph "Data Plane"
        direction TB
        subgraph "External Data Sources"
            direction TB
            HistData["Historical Market Data (72 CSV)"]:::external
            NewsFeed["News Feed API"]:::external
            DerivData["Derivatives Data (Options CSV)"]:::external
        end

        subgraph "Data Ingestion & Preprocessing"
            direction TB
            DataIngestion["Data Ingestion & Preprocessing\n(pandas, NumPy, statsmodels)"]:::ingestion
        end

        subgraph "Feature Engineering"
            direction TB
            TechInd["Technical Indicators\n(MA, EMA, MACD, Bollinger)"]:::feature
            Sentiment["Sentiment Analysis\n(BERT/Gluon)"]:::feature
            Fourier["Fourier & Trend Extraction"]:::feature
            ARIMA["ARIMA Features"]:::feature
            AutoEnc["Autoencoder Features\n(VAE + PCA)"]:::feature
            Anomaly["Anomaly Detection\n(SOM on Options)"]:::feature
            FImp["Feature Importance\n(XGBoost)"]:::feature
        end
    end

    subgraph "Model Plane"
        direction TB
        subgraph "Model Training"
            direction TB
            Generator["LSTM Generator\n(112→500→1)"]:::model
            Discriminator["1D-CNN Discriminator\n(Conv1D→BatchNorm→LeakyReLU→Dense)"]:::model
            TrainingLoop["Training Loop\n(WGAN + Metropolis-Hastings,\nCyclical LR)"]:::model
        end
    end

    subgraph "Control Plane"
        direction TB
        HyperOpt["Hyperparameter Optimization\n(RL: Rainbow DQN & PPO,\nBayesian Optimization)"]:::optimize
    end

    subgraph "Evaluation & Visualization"
        direction TB
        EvalViz["Plots & Charts\n(matplotlib, seaborn)"]:::eval
    end

    %% Data flows
    HistData -->|raw data| DataIngestion
    NewsFeed -->|news text| DataIngestion
    DerivData -->|options data| DataIngestion

    DataIngestion -->|clean data| TechInd
    DataIngestion --> Sentiment
    DataIngestion --> Fourier
    DataIngestion --> ARIMA
    DataIngestion --> AutoEnc
    DataIngestion --> Anomaly
    DataIngestion --> FImp

    TechInd -->|features| Generator
    Sentiment --> Generator
    Fourier --> Generator
    ARIMA --> Generator
    AutoEnc --> Generator
    Anomaly --> Generator
    FImp --> Generator

    Generator --> TrainingLoop
    Discriminator --> TrainingLoop
    TrainingLoop --> Discriminator
    TrainingLoop --> Generator

    TrainingLoop -->|reward| HyperOpt
    HyperOpt -->|hyperparams| TrainingLoop

    TrainingLoop -->|predictions| EvalViz
    HistData -->|ground truth| EvalViz

    %% Click Events
    click DataIngestion "https://github.com/borisbanushev/stockpredictionai/blob/master/readme.md"
    click Sentiment "https://github.com/borisbanushev/stockpredictionai/blob/master/readme2.md"
    click Generator "https://github.com/borisbanushev/stockpredictionai/blob/master/readme2.md"
    click EvalViz "https://github.com/borisbanushev/stockpredictionai/tree/master/imgs/"

    %% Styles
    classDef external fill:#a8f0c6,stroke:#333,stroke-width:1px
    classDef ingestion fill:#a8d4f0,stroke:#333,stroke-width:1px
    classDef feature fill:#a8d4f0,stroke:#333,stroke-width:1px
    classDef model fill:#f0b16b,stroke:#333,stroke-width:1px
    classDef optimize fill:#d3a0f0,stroke:#333,stroke-width:1px
    classDef eval fill:#d0d0d0,stroke:#333,stroke-width:1px
```
