How faisalm1997/kafka-ecommerce-platform works
Project Type and Purpose
• This is a full-stack real-time e-commerce analytics platform.
• Purpose: generate mock e-commerce events, stream them through Confluent Kafka, process them with Spark, ingest raw data into S3, and provide live visualization via Streamlit.
• Key technologies: Docker Compose, Confluent Kafka (broker, Schema Registry, Control Center, Connect), Apache Spark Structured Streaming, Streamlit, AWS S3.
Analysis of File Structure
• Top level:
• docker-compose.yml & docker/: orchestrates Kafka services, consumer & producer images
• kafka-connect-jars/: required JARs for Kafka Connect S3 sink
• Makefile: build, up, down, logs, processor, dashboard commands
• src/: Python modules for each component
• src/:
• config.py: centralized configuration (bootstrap servers, topics, S3 bucket names)
• data_generator.py: produces synthetic ecommerce events
• kafka_producer.py: publishes events to Kafka topics with Schema Registry integration
• kafka_consumer.py: basic consumer for event-level processing (also used to verify raw ingestion)
• spark_notebook.py: exploratory notebook for Spark queries
• spark_processor.py: Spark Structured Streaming job for real-time analytics
• streamlit_dashboard.py: Streamlit app to fetch metrics and render dashboards
Key Insights from README
• Core components: Producer, Consumer, Spark Processor, Streamlit Dashboard, Confluent Control Center.
• Data flow: events → Kafka → (1) kafka-connect → S3, (2) Spark → metrics, (3) consumer error handling.
• Services run in Docker (broker, zookeeper, schema-registry, connect, control-center).
• Local dev uses Python virtualenv; Docker containers use internal hostnames.
Main Components
a. Data Generator & Producer
• Python scripts (data_generator.py → kafka_producer.py)
• Publishes ecommerce events to Kafka topics via Schema Registry
b. Confluent Kafka Platform
• Broker(s)
• Zookeeper
• Schema Registry
• Control Center
• Kafka Connect (with S3 sink connector)
c. Storage
• AWS S3 bucket (via Kafka Connect)
d. Consumers & Processors
• kafka_consumer.py (per-event processing, error handling)
• spark_processor.py (real-time aggregation & analytics)
e. Visualization
• streamlit_dashboard.py (reads processed metrics, displays live charts)
Relationships & Data Flows
• Data Generator → Kafka Producer
• Producer → Kafka Topic “ecommerce-events”
• Kafka Topic →
• Kafka Connect S3 Sink → AWS S3 (raw event storage)
• kafka_consumer.py → custom processing / error handling
• spark_processor.py → real-time analytics (writes output to a new topic or in-memory store)
• streamlit_dashboard.py → reads analytics output (from Kafka or Spark’s in-memory sink)
• Control Center monitors all Kafka components
Architectural Patterns & Principles
• Event-driven, pub-sub streaming architecture
• Separation of concerns: producers vs. connectors vs. processors vs. visualization
• Containerization for environment parity (Docker Compose)
• Schema management via Confluent Schema Registry
• Fault tolerance: retries in Kafka Consumer, durable storage in S3
Detail Level
• Python scripts as separate components if they run independently
• Docker Compose orchestration as a container group surrounding Kafka services
• JAR files folder feeding into the Kafka Connect component
• Port mappings (e.g., broker:29092, control-center:9021, dashboard:8501)