How alexklein/kafka_data_processing works
Project type and purpose
• This is a real-time streaming data‐processing pipeline. It uses Apache Kafka as a message bus, a Python “Producer” service to generate simulated stock quotes, and a Python “Consumer” service that both writes raw data into PostgreSQL and processes it with Apache Spark. Docker Compose ties everything together.
File structure
• Top level:
• project/docker‐compose.yml – orchestrates all services (Kafka, Zookeeper, Postgres, Producer, Consumer, Spark).
• project/producer – Dockerfile + producer.py (generates stock data, publishes to Kafka topic “simulated_stock_prices”).
• project/consumer – Dockerfile + consumer.py variants (consumer_pg.py for direct Postgres writes; consumer_spark.py for Spark streaming).
• readme.md – describes components, build/run steps, and monitoring.
README for additional insights
• It names all major technologies (Kafka, Spark, PostgreSQL).
• It explains that the consumer forks into two threads: one to load raw data into Postgres, one to process via Spark with multiple sinks (console, CSV, Postgres).
Main components
a. Producer Service (Docker container running producer.py)
b. Kafka Cluster (Zookeeper + single Kafka broker, both in containers)
c. Consumer Service (Docker container running two consumer threads)
– Postgres Consumer
– Spark Consumer (pyspark streaming job)
d. Apache Spark (embedded or standalone cluster; exposes Spark UI)
e. PostgreSQL Database (data warehouse)
f. Optional: Monitoring UIs (Kafka Manager, Spark UI)
Relationships & data flows
1) Producer → Kafka topic “simulated_stock_prices”
2) Kafka → Consumer threads (two parallel pipelines)
• Pipeline A: write raw messages → PostgreSQL
• Pipeline B: ingest into Spark → process → output to console/CSV or PostgreSQL
3) Docker Compose network connects all containers.
Architectural patterns & design principles
• Microservices: decoupled Producer and Consumer services
• Event-driven streaming: Kafka topic as backbone
• Dual‐sink consumer: raw storage + analytic processing
• Containerization: Docker + Compose for reproducible environments
Key technologies & frameworks
• Python + kafka-python for Producer/Consumer
• Apache Kafka & Zookeeper
• Apache Spark (Structured Streaming)
• PostgreSQL
• Docker & Docker Compose
– From Kafka → Consumer/Postgres & Kafka → Spark
– From Spark → Postgres/Console/CSV
– Containers/services (e.g., green rectangles)
– Analytics engine (yellow hexagon or parallelogram for Spark)
• Networking overlay: show Docker network connecting all containers.