How pinildissanayaka/realtime-data-streaming-pipeline works
Project type and purpose
• This is an end-to-end real-time data engineering pipeline (full-stack ETL/streaming solution).
• Purpose: ingest user data from an external API, orchestrate via Airflow, stream with Kafka, process with Spark, and store results in Cassandra (with Postgres as intermediate storage).
File structure
• Top level:
• README.md – project overview
• requirements.txt – Python dependencies
• apache_stream.py – Kafka producer/consumer orchestration script
• fetch_db.py – script to fetch from Postgres and publish to Kafka
• log.log – pipeline log output
• checkpoint/ – Kafka stream offset/state files
• dags/Streaming_ETL.py – Airflow DAG definition for ingestion
• test/producer.py – Kafka producer for testing
• Patterns & indicators:
• “dags/” → Airflow orchestration (Directed Acyclic Graph)
• “checkpoint/” → Kafka Streams state management
• Separate scripts for ingestion (fetch_db.py), streaming (apache_stream.py) and tests
README for additional insights
• Confirms tech stack and high-level architecture:
• Data Source: Randomuser.me API
• Orchestration: Apache Airflow → fetch+persist to PostgreSQL
• Streaming: Apache Kafka + Zookeeper
• Processing: Apache Spark
• Storage: Cassandra (final), PostgreSQL (intermediate)
Key reminders
• Critical components and where they live in the file tree
• Don’t overthink—break into as many logical parts as makes sense (scripts, clusters, storage)
• Be detailed: include DAG definitions, checkpoint directories, test scripts
• The goal is a one-glance view of how data flows from source to final storage