# victor369basu/real-time-stock-market-prediction architecture

> Architecture diagram of the GitHub repository victor369basu/real-time-stock-market-prediction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/victor369basu/real-time-stock-market-prediction
- Repository: https://github.com/victor369basu/real-time-stock-market-prediction

## Overview

##### Project Type and Purpose
- This is a server‐side, real‐time data‐streaming and machine‐learning pipeline built in Node.js.
- It reads time‐series stock data (CSV files), streams it through Kafka topics, stores raw and processed data in MongoDB, trains and validates an ML model (TensorFlow.js), and performs real‐time predictions.

##### File Structure Analysis
- Top-level directories/files:
- Datasets/ – source CSVs for stock data.
- models/ – serialized TF.js model (model.json + weights.bin).
- Key scripts:
- createTopics.js – sets up Kafka topics.
- producer.js (+ start.sh) – reads CSVs & publishes records to Kafka.
- consumer.js – subscribes to “raw” topic and writes logs into MongoDB.
- ml_consumer.js – subscribes to “prediction” topic, loads saved model weights, and outputs real‐time predictions.
- tf_train.js – loads stored data from MongoDB, preprocesses (pre_process.js), trains the TF.js model (tf_model.js), validates (tf_validate.js), and saves performance metrics back to MongoDB and the models/ folder.
- InstantiateDB.js – initializes MongoDB collections/indexes.
- server.js – orchestrates or exposes any HTTP/API endpoints (if applicable).
- config.js & utils.json – centralize connection strings, topic names, and other constants.
- webpack.config.js – bundling logic, if the project ever requires front‐end assets.

##### README Insights
- Highlights a two‐topic Kafka pipeline:
1) Raw logs → MongoDB (consumer.js)
2) Raw logs → ML pipeline → real‐time predictions (ml_consumer.js)
- Describes ML workflow: 80/20 train/validation split, MinMax–Scaler preprocessing, performance chart storage in MongoDB.
- Mentions future client‐side development (React.js).

##### Main Components & Their Interactions
A. Data Source
- CSV files (Datasets/)
→ Producer (producer.js)
B. Kafka Layer
- createTopics.js defines two topics (“raw‐data”, “ml‐input”).
- Kafka Broker (and ZooKeeper) sit between producer and consumers.
C. Consumers
1. consumer.js
– Subscribes to “raw‐data” topic
– Writes incoming records into MongoDB
2. ml_consumer.js
– Subscribes to “ml‐input” topic
– Loads persisted TF.js model (models/)
– Emits real‐time predictions to stdout or downstream API
D. Data Store
- MongoDB
– Collections for raw logs, model performance metrics, and possibly preprocessed data
E. Machine Learning Workflow
- pre_process.js transforms raw logs into normalized tensors
- tf_model.js defines the model architecture
- tf_train.js executes training, saves model → models/
- tf_validate.js runs validation and stores metrics in MongoDB
F. Orchestration & Utilities
- InstantiateDB.js – one‐time DB setup
- server.js – optional HTTP layer or main orchestrator
- config.js – environment and connection settings
- utils.json – shared constants

##### Architectural Patterns & Design Principles
- Event‐driven, decoupled microservices (producer/consumers) via Kafka.
- Parallel pipelines: storage and ML prediction run concurrently.
- Separation of concerns: data ingestion, persistence, preprocessing, training, serving.
- Modular Node.js scripts—each script has a single responsibility.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Ingestion Layer
    subgraph "Ingestion" 
        direction TB
        DataSource["CSV Data Source"]:::service
        CreateTopics["Kafka Topic Setup\n(createTopics.js)"]:::kafka
        Producer["Producer Service\n(producer.js)"]:::service
        StartScript["Start Script\n(start.sh)"]:::service
        DataSource -->|"raw CSV records"| Producer
        Producer -->|"produce to topics"| CreateTopics
        CreateTopics -->|creates topics on| KafkaBroker
    end

    %% Kafka Layer
    subgraph "Kafka Cluster"
        direction TB
        KafkaBroker(["Kafka Broker\n& ZooKeeper"]):::kafka
        subgraph "Topics"
            direction TB
            RawTopic["raw-data"]:::kafka
            MLTopic["ml-input"]:::kafka
        end
        KafkaBroker --> RawTopic
        KafkaBroker --> MLTopic
    end

    %% Persistence Layer
    subgraph "Persistence"
        direction TB
        InstantiateDB["DB Initialization\n(InstantiateDB.js)"]:::util
        DBConsumer["Raw Data Consumer\n(consumer.js)"]:::service
        MongoDB["MongoDB"]:::db
        MLConsumer["ML Prediction Consumer\n(ml_consumer.js)"]:::service
        InstantiateDB -->|setup collections| MongoDB
        RawTopic -->|"raw record"| DBConsumer
        DBConsumer -->|store raw record| MongoDB
        MLTopic -->|"to prediction"| MLConsumer
        MLConsumer -->|read model & predict| ModelStore
    end

    %% ML Workflow Layer
    subgraph "ML Workflow"
        direction TB
        Preprocess["Preprocessing Module\n(pre_process.js)"]:::ml
        TFModel["Model Definition\n(tf_model.js)"]:::ml
        Train["Training Service\n(tf_train.js)"]:::ml
        Validate["Validation Service\n(tf_validate.js)"]:::ml
        ModelStore["Model Store\n(models/)"]:::ml
        MongoDB -->|fetch training data| Preprocess
        Preprocess -->|normalized tensors| TFModel
        TFModel -->|architecture| Train
        Train -->|save model weights| ModelStore
        Train -->|metrics| MongoDB
        Validate -->|validation metrics| MongoDB
    end

    %% Serving Layer
    subgraph "Serving & Orchestration"
        direction TB
        Server["HTTP/API Orchestrator\n(server.js)"]:::service
        Config["Configuration\n(config.js)"]:::util
        Utils["Shared Constants\n(utils.json)"]:::util
        Webpack["Build Config\n(webpack.config.js)"]:::util
        Package["Project Manifest\n(package.json)"]:::util
        Readme["Documentation\n(README.md)"]:::util
        Images["Illustrative Diagrams\n(images/)"]:::util
        Server -->|reads raw & metrics| MongoDB
        Server -->|serves predictions| MLConsumer
    end

    %% Click Events
    click DataSource "https://github.com/victor369basu/real-time-stock-market-prediction/tree/master/Datasets/"
    click CreateTopics "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/createTopics.js"
    click Producer "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/producer.js"
    click StartScript "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/start.sh"
    click DBConsumer "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/consumer.js"
    click MLConsumer "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/ml_consumer.js"
    click InstantiateDB "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/InstantiateDB.js"
    click Config "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/config.js"
    click Utils "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/utils.json"
    click Preprocess "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/pre_process.js"
    click TFModel "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/tf_model.js"
    click Train "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/tf_train.js"
    click Validate "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/tf_validate.js"
    click ModelStore "https://github.com/victor369basu/real-time-stock-market-prediction/tree/master/models/"
    click Server "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/server.js"
    click Webpack "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/webpack.config.js"
    click Package "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/package.json"
    click Readme "https://github.com/victor369basu/real-time-stock-market-prediction/blob/master/README.md"
    click Images "https://github.com/victor369basu/real-time-stock-market-prediction/tree/master/images/"

    %% Styles
    classDef kafka fill:#FFA500,stroke:#333,stroke-width:1px
    classDef ml fill:#90EE90,stroke:#333,stroke-width:1px
    classDef db fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef util fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef service fill:#E8E8E8,stroke:#333,stroke-width:1px
```
