# faisalm1997/kafka-ecommerce-platform architecture

> Architecture diagram of the GitHub repository faisalm1997/kafka-ecommerce-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/faisalm1997/kafka-ecommerce-platform
- Repository: https://github.com/faisalm1997/kafka-ecommerce-platform

## Overview

##### Project Type and Purpose
- This is a full-stack real-time e-commerce analytics platform.
- Purpose: generate mock e-commerce events, stream them through Confluent Kafka, process them with Spark, ingest raw data into S3, and provide live visualization via Streamlit.
- Key technologies: Docker Compose, Confluent Kafka (broker, Schema Registry, Control Center, Connect), Apache Spark Structured Streaming, Streamlit, AWS S3.

##### Analysis of File Structure
- Top level:
- docker-compose.yml & docker/: orchestrates Kafka services, consumer & producer images
- kafka-connect-jars/: required JARs for Kafka Connect S3 sink
- Makefile: build, up, down, logs, processor, dashboard commands
- src/: Python modules for each component
- src/:
- config.py: centralized configuration (bootstrap servers, topics, S3 bucket names)
- data_generator.py: produces synthetic ecommerce events
- kafka_producer.py: publishes events to Kafka topics with Schema Registry integration
- kafka_consumer.py: basic consumer for event-level processing (also used to verify raw ingestion)
- spark_notebook.py: exploratory notebook for Spark queries
- spark_processor.py: Spark Structured Streaming job for real-time analytics
- streamlit_dashboard.py: Streamlit app to fetch metrics and render dashboards

##### Key Insights from README
- Core components: Producer, Consumer, Spark Processor, Streamlit Dashboard, Confluent Control Center.
- Data flow: events → Kafka → (1) kafka-connect → S3, (2) Spark → metrics, (3) consumer error handling.
- Services run in Docker (broker, zookeeper, schema-registry, connect, control-center).
- Local dev uses Python virtualenv; Docker containers use internal hostnames.

##### Main Components
a. Data Generator & Producer
- Python scripts (data_generator.py → kafka_producer.py)
- Publishes ecommerce events to Kafka topics via Schema Registry
b. Confluent Kafka Platform
- Broker(s)
- Zookeeper
- Schema Registry
- Control Center
- Kafka Connect (with S3 sink connector)
c. Storage
- AWS S3 bucket (via Kafka Connect)
d. Consumers & Processors
- kafka_consumer.py (per-event processing, error handling)
- spark_processor.py (real-time aggregation & analytics)
e. Visualization
- streamlit_dashboard.py (reads processed metrics, displays live charts)

##### Relationships & Data Flows
- Data Generator → Kafka Producer
- Producer → Kafka Topic “ecommerce-events”
- Kafka Topic →
- Kafka Connect S3 Sink → AWS S3 (raw event storage)
- kafka_consumer.py → custom processing / error handling
- spark_processor.py → real-time analytics (writes output to a new topic or in-memory store)
- streamlit_dashboard.py → reads analytics output (from Kafka or Spark’s in-memory sink)
- Control Center monitors all Kafka components

##### Architectural Patterns & Principles
- Event-driven, pub-sub streaming architecture
- Separation of concerns: producers vs. connectors vs. processors vs. visualization
- Containerization for environment parity (Docker Compose)
- Schema management via Confluent Schema Registry
- Fault tolerance: retries in Kafka Consumer, durable storage in S3

##### Detail Level
- Python scripts as separate components if they run independently
- Docker Compose orchestration as a container group surrounding Kafka services
- JAR files folder feeding into the Kafka Connect component
- Port mappings (e.g., broker:29092, control-center:9021, dashboard:8501)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Ingestion Layer
    subgraph "Ingestion Layer"
        Config["Config (config.py)"]:::ingestion
        DataGen["Data Generator (data_generator.py)"]:::ingestion
        Producer["Kafka Producer (kafka_producer.py)"]:::ingestion
        subgraph "Docker Images"
            ProdImage["Docker Producer Image (Dockerfile.producer)"]:::ingestion
            ConsImage["Docker Consumer Image (Dockerfile.consumer)"]:::ingestion
        end
    end

    %% Messaging Layer
    subgraph "Messaging Layer"
        DC["docker-compose.yml"]:::messaging
        subgraph "Kafka Cluster Services"
            Broker["Kafka Broker"]:::messaging
            ZK["Zookeeper"]:::messaging
            SchemaReg["Schema Registry"]:::messaging
            Connect["Kafka Connect"]:::messaging
            ControlCenter["Control Center"]:::messaging
        end
        Jars["Kafka Connect JARs"]:::messaging
    end

    %% Storage Layer
    subgraph "Storage Layer"
        S3[(AWS S3 Bucket)]:::storage
    end

    %% Processing Layer
    subgraph "Processing Layer"
        Consumer["Kafka Consumer (kafka_consumer.py)"]:::processing
        SparkProc["Spark Processor (spark_processor.py)"]:::processing
        Notebook["Spark Notebook (spark_notebook.py)"]:::processing
    end

    %% Presentation Layer
    subgraph "Presentation Layer"
        Dashboard["Streamlit Dashboard (streamlit_dashboard.py)"]:::presentation
    end

    %% Project Artifacts
    subgraph "Project Artifacts"
        Makefile["Makefile"]:::artifacts
        Req["requirements.txt"]:::artifacts
    end

    %% Connections
    Config --> DataGen
    Config --> Producer
    DataGen --> Producer
    Producer --> Broker
    DC --> Broker
    DC --> ZK
    DC --> SchemaReg
    DC --> Connect
    DC --> ControlCenter
    Jars --> Connect
    Broker --> Connect
    Connect --> S3
    Broker --> Consumer
    Broker --> SparkProc
    Broker --> Notebook
    SparkProc --> Dashboard

    %% Click Events
    click DataGen "https://github.com/faisalm1997/kafka-ecommerce-platform/blob/main/src/data_generator.py"
    click Producer "https://github.com/faisalm1997/kafka-ecommerce-platform/blob/main/src/kafka_producer.py"
    click Config "https://github.com/faisalm1997/kafka-ecommerce-platform/blob/main/src/config.py"
    click Consumer "https://github.com/faisalm1997/kafka-ecommerce-platform/blob/main/src/kafka_consumer.py"
    click SparkProc "https://github.com/faisalm1997/kafka-ecommerce-platform/blob/main/src/spark_processor.py"
    click Notebook "https://github.com/faisalm1997/kafka-ecommerce-platform/blob/main/src/spark_notebook.py"
    click Dashboard "https://github.com/faisalm1997/kafka-ecommerce-platform/blob/main/src/streamlit_dashboard.py"
    click DC "https://github.com/faisalm1997/kafka-ecommerce-platform/blob/main/docker-compose.yml"
    click ProdImage "https://github.com/faisalm1997/kafka-ecommerce-platform/blob/main/docker/Dockerfile.producer"
    click ConsImage "https://github.com/faisalm1997/kafka-ecommerce-platform/blob/main/docker/Dockerfile.consumer"
    click Jars "https://github.com/faisalm1997/kafka-ecommerce-platform/tree/main/kafka-connect-jars/"
    click Makefile "https://github.com/faisalm1997/kafka-ecommerce-platform/tree/main/Makefile"
    click Req "https://github.com/faisalm1997/kafka-ecommerce-platform/blob/main/requirements.txt"

    %% Styles
    classDef ingestion fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef messaging fill:#98FB98,stroke:#000,stroke-width:1px
    classDef processing fill:#FFA500,stroke:#000,stroke-width:1px
    classDef storage fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef presentation fill:#D8BFD8,stroke:#000,stroke-width:1px
    classDef artifacts fill:#FFFFFF,stroke:#000,stroke-width:1px,stroke-dasharray: 5 5
```
