# arjun-n2603/emostream architecture

> Architecture diagram of the GitHub repository arjun-n2603/emostream, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arjun-n2603/emostream
- Repository: https://github.com/arjun-n2603/emostream

## Overview

##### Project Type and Purpose
- EmoStream is a backend-focused, event-driven streaming system (no dedicated frontend code in the repo).
- Purpose: capture high-volume emoji reactions via REST, queue them in Kafka, process in Spark Streaming, then publish aggregated results to many clients via a pub-sub architecture.

##### Key Components (map files → logical services)
a. Emoji Ingestion API
- main_server.py (Flask/Express endpoint)
- emojiproducer.py (Kafka producer for test/demo or auxiliary ingestion)
b. Message Broker
- Kafka cluster (topics for raw emojis and aggregated outputs)
c. Stream Processing
- spark_backend.py (Spark Streaming job reading raw topic, aggregating emojis in 2s micro-batches)
d. Aggregator Publisher
- mainpublisher.py (Kafka producer pushing aggregated results to “aggregated” topic)
e. Cluster Publishers & Subscribers
- clusterXproducer.py (X=1,2,3): read aggregated topic, produce to cluster-specific topic (for load-segmentation)
- clusterXserverY.py (Y=1..3): subscriber servers per cluster serving connected clients
f. Clients
- Mobile/web apps subscribing via WebSocket/HTTP to cluster servers

##### Relationships & Data Flow
- Clients → Emoji Ingestion API → Kafka “raw-emojis” topic
- Spark Streaming consumes “raw-emojis” → applies aggregation logic → writes to “aggregated-emojis” topic
- mainpublisher → reads “aggregated-emojis” → writes to central Kafka aggregator topic
- clusterXproducer → reads central aggregator topic → writes to “cluster-X” topic
- clusterXserverY → consumes “cluster-X” topics → pushes updates to its connected clients

##### Architectural Patterns & Principles
- Event-Driven Architecture using Kafka as durable message bus
- Micro-batch Processing (Spark Streaming)
- Pub-Sub Pattern for scalable, fan-out distribution
- Horizontal scaling via clusters (3 clusters × 3 servers each)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Ingestion Layer
    subgraph "Ingestion Layer"
        direction TB
        MS["main_server.py\n(Flask REST)"]:::ingestion
        EP["emojiproducer.py"]:::ingestion
        RAW["raw-emojis\nKafka Topic"]:::kafka
    end

    %% Processing Layer
    subgraph "Processing Layer"
        direction TB
        SP["spark_backend.py\nSpark Streaming\n(2s micro-batch)"]:::processing
        AGG["aggregated-emojis\nKafka Topic"]:::kafka
    end

    %% Aggregator Publisher
    subgraph "Aggregator Publisher"
        direction TB
        MP["mainpublisher.py"]:::distribution
        CENTRAL["aggregated-stream\nKafka Topic"]:::kafka
    end

    %% Cluster Distribution
    subgraph "Cluster Distribution"
        direction TB

        subgraph "Cluster 1"
            direction TB
            C1P["cluster1producer.py"]:::distribution
            C1T["cluster-1\nKafka Topic"]:::kafka
            C1S1["cluster1server1.py"]:::distribution
            C1S2["cluster1server2.py"]:::distribution
            C1S3["cluster1server3.py"]:::distribution
        end

        subgraph "Cluster 2"
            direction TB
            C2P["cluster2producer.py"]:::distribution
            C2T["cluster-2\nKafka Topic"]:::kafka
            C2S1["cluster2server1.py"]:::distribution
            C2S2["cluster2server2.py"]:::distribution
            C2S3["cluster2server3.py"]:::distribution
        end

        subgraph "Cluster 3"
            direction TB
            C3P["cluster3producer.py"]:::distribution
            C3T["cluster-3\nKafka Topic"]:::kafka
            C3S1["cluster3server1.py"]:::distribution
            C3S2["cluster3server2.py"]:::distribution
            C3S3["cluster3server3.py"]:::distribution
        end
    end

    %% Clients
    CLIENTS["Clients\n(Mobile/Web)"]:::external

    %% Data Flow
    CLIENTS -->|"POST emoji via REST"| MS
    MS -->|"raw-emojis"| RAW
    EP -->|"raw-emojis"| RAW
    RAW -->|"consume raw-emojis"| SP
    SP -->|"aggregated-emojis"| AGG
    AGG -->|"consume aggregated-emojis"| MP
    MP -->|"aggregated-stream"| CENTRAL

    CENTRAL -->|"consume aggregated-stream"| C1P
    C1P -->|"cluster-1"| C1T
    C1T -->|"push updates"| C1S1
    C1T -->|"push updates"| C1S2
    C1T -->|"push updates"| C1S3
    C1S1 -->|"WebSocket/HTTP"| CLIENTS
    C1S2 -->|"WebSocket/HTTP"| CLIENTS
    C1S3 -->|"WebSocket/HTTP"| CLIENTS

    CENTRAL -->|"consume aggregated-stream"| C2P
    C2P -->|"cluster-2"| C2T
    C2T -->|"push updates"| C2S1
    C2T -->|"push updates"| C2S2
    C2T -->|"push updates"| C2S3
    C2S1 -->|"WebSocket/HTTP"| CLIENTS
    C2S2 -->|"WebSocket/HTTP"| CLIENTS
    C2S3 -->|"WebSocket/HTTP"| CLIENTS

    CENTRAL -->|"consume aggregated-stream"| C3P
    C3P -->|"cluster-3"| C3T
    C3T -->|"push updates"| C3S1
    C3T -->|"push updates"| C3S2
    C3T -->|"push updates"| C3S3
    C3S1 -->|"WebSocket/HTTP"| CLIENTS
    C3S2 -->|"WebSocket/HTTP"| CLIENTS
    C3S3 -->|"WebSocket/HTTP"| CLIENTS

    %% Click Events
    click MS "https://github.com/arjun-n2603/emostream/blob/main/main_server.py"
    click EP "https://github.com/arjun-n2603/emostream/blob/main/emojiproducer.py"
    click SP "https://github.com/arjun-n2603/emostream/blob/main/spark_backend.py"
    click MP "https://github.com/arjun-n2603/emostream/blob/main/mainpublisher.py"
    click C1P "https://github.com/arjun-n2603/emostream/blob/main/cluster1producer.py"
    click C2P "https://github.com/arjun-n2603/emostream/blob/main/cluster2producer.py"
    click C3P "https://github.com/arjun-n2603/emostream/blob/main/cluster3producer.py"
    click C1S1 "https://github.com/arjun-n2603/emostream/blob/main/cluster1server1.py"
    click C1S2 "https://github.com/arjun-n2603/emostream/blob/main/cluster1server2.py"
    click C1S3 "https://github.com/arjun-n2603/emostream/blob/main/cluster1server3.py"
    click C2S1 "https://github.com/arjun-n2603/emostream/blob/main/cluster2server1.py"
    click C2S2 "https://github.com/arjun-n2603/emostream/blob/main/cluster2server2.py"
    click C2S3 "https://github.com/arjun-n2603/emostream/blob/main/cluster2server3.py"
    click C3S1 "https://github.com/arjun-n2603/emostream/blob/main/cluster3server1.py"
    click C3S2 "https://github.com/arjun-n2603/emostream/blob/main/cluster3server2.py"
    click C3S3 "https://github.com/arjun-n2603/emostream/blob/main/cluster3server3.py"

    %% Styles
    classDef ingestion fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef processing fill:#90EE90,stroke:#000,stroke-width:1px
    classDef distribution fill:#FFD580,stroke:#000,stroke-width:1px
    classDef kafka fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef external fill:#F0F0F0,stroke:#000,stroke-width:1px
```
