How airscholar/e2e-data-engineering works
Project Type and Purpose
• This is an end-to-end, containerized, real-time data engineering pipeline.
• Purpose: Ingest random user data, orchestrate with Airflow, stream via Kafka/Zookeeper, process in Spark, and persist in Cassandra (with Postgres as intermediate store).
File Structure
• Top-level files:
• docker-compose.yml – brings up all containers (Airflow, Kafka, Zookeeper, Spark, Cassandra, Postgres, Control Center, Schema Registry).
• requirements.txt – Python dependencies for DAGs and stream scripts.
• spark_stream.py – Spark consumer job.
• dags/
• kafka_stream.py – Airflow DAG that fetches API data into Postgres and publishes to Kafka.
• script/entrypoint.sh – Container entrypoint for initializing services (likely used for Kafka/Zookeeper or Airflow).
• .gitignore, README.md – boilerplate.
README for Additional Insights
• Components listed: Data Source, Airflow, Postgres, Kafka & Zookeeper, Control Center & Schema Registry, Spark (master & workers), Cassandra.
• All services are defined in docker-compose.yml for local orchestration.
a. Main Components:
• External Data Source: randomuser.me API
• Orchestration: Apache Airflow (Scheduler + Webserver + Worker)
• Relational Store: PostgreSQL (intermediate landing)
• Streaming Layer: Apache Kafka + Zookeeper
• Management: Confluent Control Center + Schema Registry
• Compute: Apache Spark cluster (Master + Workers running spark_stream.py)
• Final Storage: Apache Cassandra
• Docker Network: overarching container network
b. Relationships & Data Flow:
1. Airflow DAG → HTTP fetch → writes raw JSON to Postgres
2. Airflow (or a Kafka producer task) → reads from Postgres → publishes to Kafka topic
3. Zookeeper manages Kafka brokers; Schema Registry validates Avro/JSON schemas
4. Spark Streaming job (spark_stream.py) → consumes Kafka topic → transforms data
5. Spark → writes processed records to Cassandra
c. Architectural Patterns & Principles:
• Orchestration pattern (Airflow DAGs)
• Lambda-style streaming (micro-batch via Spark Streaming)
• Event-driven architecture (Kafka topics as event bus)
• Containerized microservices via Docker Compose
d. Key Technologies/Frameworks:
• Python (DAGs, producers)
• Apache Airflow for orchestration
• PostgreSQL as staging store
• Apache Kafka & Zookeeper for durable streaming
• Confluent Control Center & Schema Registry for monitoring/schemas
• Apache Spark for distributed processing
• Cassandra for NoSQL storage
• Docker Compose for orchestration of all services