How victor369basu/real-time-stock-market-prediction works
Project Type and Purpose
• This is a server‐side, real‐time data‐streaming and machine‐learning pipeline built in Node.js.
• It reads time‐series stock data (CSV files), streams it through Kafka topics, stores raw and processed data in MongoDB, trains and validates an ML model (TensorFlow.js), and performs real‐time predictions.
File Structure Analysis
• Top-level directories/files:
• Datasets/ – source CSVs for stock data.
• models/ – serialized TF.js model (model.json + weights.bin).
• Key scripts:
• createTopics.js – sets up Kafka topics.
• producer.js (+ start.sh) – reads CSVs & publishes records to Kafka.
• consumer.js – subscribes to “raw” topic and writes logs into MongoDB.
• ml_consumer.js – subscribes to “prediction” topic, loads saved model weights, and outputs real‐time predictions.
• tf_train.js – loads stored data from MongoDB, preprocesses (pre_process.js), trains the TF.js model (tf_model.js), validates (tf_validate.js), and saves performance metrics back to MongoDB and the models/ folder.
• InstantiateDB.js – initializes MongoDB collections/indexes.
• server.js – orchestrates or exposes any HTTP/API endpoints (if applicable).
• config.js & utils.json – centralize connection strings, topic names, and other constants.
• webpack.config.js – bundling logic, if the project ever requires front‐end assets.
README Insights
• Highlights a two‐topic Kafka pipeline:
1) Raw logs → MongoDB (consumer.js)
2) Raw logs → ML pipeline → real‐time predictions (ml_consumer.js)
• Describes ML workflow: 80/20 train/validation split, MinMax–Scaler preprocessing, performance chart storage in MongoDB.
• Mentions future client‐side development (React.js).
Main Components & Their Interactions
A. Data Source
• CSV files (Datasets/)
→ Producer (producer.js)
B. Kafka Layer
• createTopics.js defines two topics (“raw‐data”, “ml‐input”).
• Kafka Broker (and ZooKeeper) sit between producer and consumers.
C. Consumers
1. consumer.js
– Subscribes to “raw‐data” topic
– Writes incoming records into MongoDB
2. ml_consumer.js
– Subscribes to “ml‐input” topic
– Loads persisted TF.js model (models/)
– Emits real‐time predictions to stdout or downstream API
D. Data Store
• MongoDB
– Collections for raw logs, model performance metrics, and possibly preprocessed data
E. Machine Learning Workflow
• pre_process.js transforms raw logs into normalized tensors
• tf_model.js defines the model architecture
• tf_train.js executes training, saves model → models/
• tf_validate.js runs validation and stores metrics in MongoDB
F. Orchestration & Utilities
• InstantiateDB.js – one‐time DB setup
• server.js – optional HTTP layer or main orchestrator
• config.js – environment and connection settings
• utils.json – shared constants
Architectural Patterns & Design Principles
• Event‐driven, decoupled microservices (producer/consumers) via Kafka.
• Parallel pipelines: storage and ML prediction run concurrently.
• Separation of concerns: data ingestion, persistence, preprocessing, training, serving.
• Modular Node.js scripts—each script has a single responsibility.