How pranav-8660/realtimedatastreamer-apachekafka works
Project Type and Purpose
• Type: Real-time data streaming pipeline.
• Purpose: Generate synthetic sensor data, stream it through Kafka, and persist it into PostgreSQL for downstream analysis.
File Structure
• Top-level:
– kafka-producer/
• data_producer.py (publishes JSON messages to Kafka topic)
• venv/ (Python virtual environment for isolation)
– kafka-consumer/
• kafka_consumer.py (subscribes to Kafka topic, writes to Postgres)
– venv/ (root-level environment, alternative setup)
– commands.txt (CLI instructions, optional)
• No web UI—pure data pipeline.
Extract Key Components from README
• Producer side:
– Python script using kafka-python, uuid, python-dotenv
– Kafka topic “sensor-data” on a Kafka broker (localhost:9092)
• Messaging layer:
– Zookeeper (coordination), Kafka broker, topic partitions
• Consumer side:
– Python script using kafka-python, psycopg2-binary
– PostgreSQL 14+ database “sensor_data” with table sensor_readings
• Configuration via .env or environment variables
a. Data Producer
• Tech: Python + kafka-python
b. Kafka Cluster
• Annotation: Partitions = 1, Replication factor = 1
c. Data Consumer
• Tech: Python + kafka-python + psycopg2
d. PostgreSQL Database
e. (Optional) CLI/Monitoring
Architectural Patterns & Principles
• Asynchronous, decoupled pipeline (pub/sub)
• Single responsibility: each script handles one step
• Config via environment variables (12-factor)
• Fault-tolerant messaging (Kafka) and durable storage (Postgres)