How kdb04/tweeeeedbt works
Project type and purpose
• This is a real-time data streaming pipeline (“TweeeeDBT”) built in Python that uses Apache Kafka for ingestion, Apache Spark for stream processing, and PostgreSQL as the persistent store.
• Its purpose is to simulate tweet data production, process/transform it in real time, and persist various views (geolocation, team mentions, verified-user windows, etc.) into a database.
File structure
• Top-level scripts:
• producer.py – generates/sends raw tweet events into Kafka
• stream_processing.py – a Spark Structured Streaming job that reads raw events from one or more Kafka topics, transforms/enriches them, and writes results back to other Kafka topics
• batch_processing.py – an offline Spark job for batch analytics (e.g., historical aggregations)
• consumer_<…>.py – separate Python consumers for each processed topic (geolocation, team mentions, user verification streams, windowed verified user counts) that read from Kafka and write into PostgreSQL
• Supporting files:
• docker-compose.yaml – spins up Zookeeper and Kafka containers
• reset-kafka.sh – resets Kafka topics between runs
• DB/ – contains schema.sql (PostgreSQL DDL) and dbcheck.py (sanity-check script)
• requirements.txt – Python dependencies
• No front-end; purely back-end streaming micro-services.
README for additional insights
• Shows how to launch Zookeeper/Kafka via Docker Compose
• Spark standalone set-up instructions (local Spark)
• PostgreSQL installation, user creation, schema application
• Run order:
1. reset-kafka.sh
2. spark-submit stream_processing.py
3. python3 consumer_*.py (each in its own terminal)
4. python3 producer.py
Main components and their relationships
a. Producer service
• publishes raw events to “tweets_raw” Kafka topic
b. Kafka cluster (Zookeeper + Kafka brokers)
• transient messaging bus between producers, stream processors, and consumers
c. Spark Streaming job
• consumes from “tweets_raw”
• applies transformations/enrichments
• writes multiple processed streams to distinct Kafka topics (e.g., “tweets_geo”, “tweets_teammentions”, “tweets_userverify”, “tweets_verified_user_windowed”)
d. Consumer services
• one micro-service per topic
• consumes enriched events and writes into respective tables in PostgreSQL
e. PostgreSQL database
• persistent store for downstream analytics or dashboards
Architectural patterns and design principles
• Separation of concerns: ingestion (producer), processing (Spark), storage (consumers + DB)
• Micro-service approach for each downstream view via separate consumer scripts
• Use of a message-broker to decouple producers, processors, and consumers
• Idempotent, stateless stream processing in Spark