# znitamohamed4/real-time-sentiment-analysis architecture

> Architecture diagram of the GitHub repository znitamohamed4/real-time-sentiment-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/znitamohamed4/real-time-sentiment-analysis
- Repository: https://github.com/znitamohamed4/real-time-sentiment-analysis

## Overview

##### Project Type and Purpose
- This is a real-time, full-stack streaming data pipeline for sentiment analysis on Amazon reviews.
- It ingests review JSONs, pipes them through Kafka, runs inference via Spark Structured Streaming with a pre-trained Spark ML model, persists predictions in MongoDB, and visualizes results through a Flask + Socket.IO dashboard.

##### Top-Level Components (based on file structure & README)
a. Data Source
- `data/` directory holds sample JSON reviews (test_data.json).
- `kafka/producer/` contains `kafka_producer.py` to read JSON and publish to Kafka.
b. Messaging Layer
- Kafka Broker (`docker/kafka/`)
- Zookeeper (`docker/zookeeper/`)
c. Stream Processor
- Spark Consumer script (`kafka/consumer/spark_consumer.py`)
- Spark master provided via Docker Compose (`docker/spark/`)
- Pre-trained Spark ML pipeline model under `model/best_model/SentimentModel/`
d. Storage
- MongoDB (`docker/mongodb/`) stores inference results
- Accessed via `backend/app/db.py`
e. Presentation Layer
- Flask app in `backend/app/` with Socket.IO for pushing live updates
- Static assets (`css/style.css`) and templates for charts (`index.html`, `layout.html`)
f. Orchestration
- `docker-compose.yml` tying together Kafka, Zookeeper, Spark, MongoDB, Flask, producer, and consumer

##### Data Flow & Interactions
1. Producer reads JSON from `data/` → publishes to Kafka input topic.
2. Kafka Broker (with Zookeeper) buffers streaming events.
3. Spark Structured Streaming consumer subscribes to Kafka topic → applies the Spark ML pipeline stages (tokenizer → stop-words remover → TF → IDF → indexer → RandomForestClassifier) → produces sentiment labels.
4. Spark job writes results to MongoDB collection.
5. Flask backend queries MongoDB or listens for change streams → emits events via WebSockets to the front-end.
6. Front-end charts update in real time.

##### Architectural Patterns & Principles
- Event-driven, decoupled microservices: producer, broker, consumer, database, dashboard.
- Single Responsibility: each container / service owns one function.
- Polyglot orchestration with Docker Compose for reproducibility.

## Diagram (Mermaid)

```mermaid
graph TB
    Data["test_data.json"]:::doc
    Producer("Kafka Producer"):::kafka
    subgraph "Docker Compose" 
        subgraph "Kafka Ecosystem"
            ZK("Zookeeper"):::kafka
            Broker("Kafka Broker"):::kafka
            ZK --> Broker
        end
        Producer --> Broker
        subgraph "Spark Processing"
            SparkMaster("Spark Master"):::spark
            Consumer("Spark Consumer"):::spark
            Model["SentimentModel"]:::doc
            Broker --> Consumer
            Consumer --> Model
        end
        subgraph "MongoDB Storage"
            MongoDB[(MongoDB)]:::db
        end
        Consumer -->|"sentiment-results"| MongoDB
        subgraph "Flask Application"
            App("Flask App"):::backend
            DBClient("db.py"):::backend
            Routes("routes.py"):::backend
            Events("events.py"):::backend
            MongoDB --> App
            App --> DBClient
            App --> Routes
            App --> Events
        end
    end
    Data --> Producer
    App -->|"Socket.IO"| Browser("Browser"):::frontend

    %% Click Events
    click Data "https://github.com/znitamohamed4/real-time-sentiment-analysis/tree/master/data/"
    click Producer "https://github.com/znitamohamed4/real-time-sentiment-analysis/blob/master/kafka/producer/kafka_producer.py"
    click ZK "https://github.com/znitamohamed4/real-time-sentiment-analysis/tree/master/docker/zookeeper/Dockerfile"
    click Broker "https://github.com/znitamohamed4/real-time-sentiment-analysis/blob/master/docker/kafka/server.properties"
    click Consumer "https://github.com/znitamohamed4/real-time-sentiment-analysis/blob/master/kafka/consumer/spark_consumer.py"
    click SparkMaster "https://github.com/znitamohamed4/real-time-sentiment-analysis/tree/master/docker/spark/Dockerfile"
    click Model "https://github.com/znitamohamed4/real-time-sentiment-analysis/tree/master/model/best_model/SentimentModel/"
    click MongoDB "https://github.com/znitamohamed4/real-time-sentiment-analysis/tree/master/docker/mongodb/Dockerfile"
    click DBClient "https://github.com/znitamohamed4/real-time-sentiment-analysis/blob/master/backend/app/db.py"
    click App "https://github.com/znitamohamed4/real-time-sentiment-analysis/blob/master/backend/run.py"
    click Routes "https://github.com/znitamohamed4/real-time-sentiment-analysis/blob/master/backend/app/routes.py"
    click Events "https://github.com/znitamohamed4/real-time-sentiment-analysis/blob/master/backend/app/events.py"

    classDef kafka fill:#FFA500,stroke:#333,stroke-width:1px
    classDef spark fill:#1E90FF,stroke:#333,stroke-width:1px
    classDef db fill:#32CD32,stroke:#333,stroke-width:1px
    classDef backend fill:#9370DB,stroke:#333,stroke-width:1px
    classDef frontend fill:#FF69B4,stroke:#333,stroke-width:1px
    classDef doc fill:#FFFACD,stroke:#333,stroke-width:1px
    classDef boundary fill:none,stroke:#000,stroke-dasharray: 5 5,stroke-width:2px
```
