How arjun-n2603/emostream works
Project Type and Purpose
• EmoStream is a backend-focused, event-driven streaming system (no dedicated frontend code in the repo).
• Purpose: capture high-volume emoji reactions via REST, queue them in Kafka, process in Spark Streaming, then publish aggregated results to many clients via a pub-sub architecture.
Key Components (map files → logical services)
a. Emoji Ingestion API
• main_server.py (Flask/Express endpoint)
• emojiproducer.py (Kafka producer for test/demo or auxiliary ingestion)
b. Message Broker
• Kafka cluster (topics for raw emojis and aggregated outputs)
c. Stream Processing
• spark_backend.py (Spark Streaming job reading raw topic, aggregating emojis in 2s micro-batches)
d. Aggregator Publisher
• mainpublisher.py (Kafka producer pushing aggregated results to “aggregated” topic)
e. Cluster Publishers & Subscribers
• clusterXproducer.py (X=1,2,3): read aggregated topic, produce to cluster-specific topic (for load-segmentation)
• clusterXserverY.py (Y=1..3): subscriber servers per cluster serving connected clients
f. Clients
• Mobile/web apps subscribing via WebSocket/HTTP to cluster servers
Relationships & Data Flow
• Clients → Emoji Ingestion API → Kafka “raw-emojis” topic
• Spark Streaming consumes “raw-emojis” → applies aggregation logic → writes to “aggregated-emojis” topic
• mainpublisher → reads “aggregated-emojis” → writes to central Kafka aggregator topic
• clusterXproducer → reads central aggregator topic → writes to “cluster-X” topic
• clusterXserverY → consumes “cluster-X” topics → pushes updates to its connected clients
Architectural Patterns & Principles
• Event-Driven Architecture using Kafka as durable message bus
• Micro-batch Processing (Spark Streaming)
• Pub-Sub Pattern for scalable, fan-out distribution
• Horizontal scaling via clusters (3 clusters × 3 servers each)