How chinmay13579/real_time_data_ingestion works
Project Type and Purpose
• This is a real-time streaming data ingestion pipeline.
• Purpose: simulate order and payment events, join them in flight, and persist the merged records into a Cassandra “fact” table.
File Structure
• Top level scripts:
• order_data_producer.py
• payments_data_producer.py
• order_data_consumer.py
• ingest_in_fact_table.py
• Infrastructure definition:
• docker-compose-cassandra.yml (brings up Cassandra, maybe Zookeeper/Kafka in other compose file or separate)
• Supporting docs:
• README.md
• setup_and_requirement_readme.txt
• No explicit “src”/“lib” subdirectories: each script is a standalone component.
README Insights
• Two Kafka producers to simulate two event streams (orders, payments).
• One Kafka consumer that joins streams and passes merged payloads to Cassandra.
• Uses Apache Kafka + Zookeeper for messaging, Cassandra for storage.
• Python 3.8+, kafka-python, cassandra-driver.
a. Main Components
• Order Producer service
• Payment Producer service
• Kafka Cluster (Brokers + Zookeeper)
• Order+Payment Consumer (stream join)
• Cassandra cluster (fact table storage)
b. Relationships and Interactions
1. Order Producer → publishes to “orders” Kafka topic
2. Payment Producer → publishes to “payments” Kafka topic
3. Consumer subscribes to both topics, joins events by key (e.g., order_id)
4. Consumer writes merged events into Cassandra fact table
c. Architectural Patterns & Principles
• Streaming ETL pattern
• Loose coupling via message bus (Kafka topics)
• Idempotent writes into Cassandra (fact table)
• Docker-based deployment for infra components
d. Relevant Technologies
• kafka-python library for producers/consumer
• cassandra-driver for DB IO
• Docker Compose for Cassandra (and possibly Kafka)
• Python 3.8 runtime
Elements
• Clear labels for each component (service or infra)
• Producers → Kafka topics → Consumer → Cassandra
• Rectangles for application services (blue)
• Annotations for message topics (orders, payments) and fact table name.
Detailed Guidance
• Optionally, show Docker Compose orchestration around Kafka & Cassandra to indicate containerized deployment.