How znitamohamed4/real-time-sentiment-analysis works
Project Type and Purpose
• This is a real-time, full-stack streaming data pipeline for sentiment analysis on Amazon reviews.
• It ingests review JSONs, pipes them through Kafka, runs inference via Spark Structured Streaming with a pre-trained Spark ML model, persists predictions in MongoDB, and visualizes results through a Flask + Socket.IO dashboard.
Top-Level Components (based on file structure & README)
a. Data Source
• data/ directory holds sample JSON reviews (test_data.json).
• kafka/producer/ contains kafka_producer.py to read JSON and publish to Kafka.
b. Messaging Layer
• Kafka Broker (docker/kafka/)
• Zookeeper (docker/zookeeper/)
c. Stream Processor
• Spark Consumer script (kafka/consumer/spark_consumer.py)
• Spark master provided via Docker Compose (docker/spark/)
• Pre-trained Spark ML pipeline model under model/best_model/SentimentModel/
d. Storage
• MongoDB (docker/mongodb/) stores inference results
• Accessed via backend/app/db.py
e. Presentation Layer
• Flask app in backend/app/ with Socket.IO for pushing live updates
• Static assets (css/style.css) and templates for charts (index.html, layout.html)
f. Orchestration
• docker-compose.yml tying together Kafka, Zookeeper, Spark, MongoDB, Flask, producer, and consumer
Data Flow & Interactions
1. Producer reads JSON from data/ → publishes to Kafka input topic.
2. Kafka Broker (with Zookeeper) buffers streaming events.
3. Spark Structured Streaming consumer subscribes to Kafka topic → applies the Spark ML pipeline stages (tokenizer → stop-words remover → TF → IDF → indexer → RandomForestClassifier) → produces sentiment labels.
4. Spark job writes results to MongoDB collection.
5. Flask backend queries MongoDB or listens for change streams → emits events via WebSockets to the front-end.
6. Front-end charts update in real time.
Architectural Patterns & Principles
• Event-driven, decoupled microservices: producer, broker, consumer, database, dashboard.
• Single Responsibility: each container / service owns one function.
• Polyglot orchestration with Docker Compose for reproducibility.