How borisbanushev/stockpredictionai works
Project type and purpose
• This is a research‐oriented data science/ML project implemented as a Jupyter notebook.
• Purpose: ingest multiple sources of time-series and textual data, engineer features (technical indicators, sentiment via BERT, Fourier/ARIMA transforms, autoencoder features, anomaly scores), then train a GAN (LSTM generator + 1D-CNN discriminator) to predict future stock price movements. A reinforcement-learning + Bayesian optimization loop tunes hyperparameters.
Main components
• Historical market data (72 correlated assets CSV)
• News feed (daily GS news for BERT-based sentiment)
• Derivatives data (options prices)
• CSV reader, date parser
• Statistical checks (heteroskedasticity, multicollinearity, serial correlation)
1. Technical indicators (MA, EMA, MACD, Bollinger)
2. Fundamental sentiment (BERT/NLP)
3. Fourier transforms & trend extraction
4. ARIMA feature
5. Autoencoder features (stacked VAE + PCA)
6. Anomaly detection (Self-Organizing Maps on options pricing)
7. Feature importance (XGBoost)
1. GAN environment
• Generator: one-layer LSTM (112 input features → 500 hidden → 1 output)
• Discriminator: 1D-CNN (Conv1D → BatchNorm → LeakyReLU → Dense → output)
• Loss: WGAN + Metropolis-Hastings sampling
2. Training loop: alternating D/G updates, learning-rate scheduler (triangular/cyclical)
• RL agent (Rainbow DQN & PPO) interacts with the GAN training environment
• Reward function = 2*G_loss + D_loss + G_accuracy
• Bayesian Optimization (Gaussian process) suggests next hyperparams
• Feedback loop into model training
• Plot real vs. predicted prices after 1, 50, 200 epochs
• Final performance charts
Relationships and Interactions
• Data Sources → Data Ingestion
• Data Ingestion → Feature Engineering
• Feature Engineering → Model Training
• Training produces a reward
• Optimizer returns new hyperparameter settings
• Model Training → Evaluation & Visualization
Architectural patterns and design principles
• Pipeline/Micro-batch pattern for sequential data processing steps
• Modular separation: data, features, model, tuning, and results
• Feedback loops: hyperparameter tuner → training → reward
• Use of asynchronous or batched updates in GAN training
• Reinforcement Learning as controller for continuous optimization
Technologies, frameworks, libraries
• Data processing: pandas, NumPy, statsmodels, scikit-learn
• Feature engineering: fft (NumPy), ARIMA (statsmodels), PCA (scikit-learn), XGBoost
• NLP: BERT (MXNet/Gluon)
• Deep learning: MXNet/Gluon (LSTM, Variational AE, Conv1D)
• GAN improvements: WGAN, Metropolis-Hastings
• Hyperparameter tuning: bayes_opt (BayesianOptimization), RL (OpenAI baselines for PPO, custom Rainbow)
• Visualization: matplotlib, seaborn