# airscholar/e2e-data-engineering architecture

> Architecture diagram of the GitHub repository airscholar/e2e-data-engineering, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/airscholar/e2e-data-engineering
- Repository: https://github.com/airscholar/e2e-data-engineering

## Overview

##### Project Type and Purpose
- This is an end-to-end, containerized, real-time data engineering pipeline.
- Purpose: Ingest random user data, orchestrate with Airflow, stream via Kafka/Zookeeper, process in Spark, and persist in Cassandra (with Postgres as intermediate store).

##### File Structure
- Top-level files:
- docker-compose.yml – brings up all containers (Airflow, Kafka, Zookeeper, Spark, Cassandra, Postgres, Control Center, Schema Registry).
- requirements.txt – Python dependencies for DAGs and stream scripts.
- spark_stream.py – Spark consumer job.
- dags/
- kafka_stream.py – Airflow DAG that fetches API data into Postgres and publishes to Kafka.
- script/entrypoint.sh – Container entrypoint for initializing services (likely used for Kafka/Zookeeper or Airflow).
- .gitignore, README.md – boilerplate.

##### README for Additional Insights
- Components listed: Data Source, Airflow, Postgres, Kafka & Zookeeper, Control Center & Schema Registry, Spark (master & workers), Cassandra.
- All services are defined in docker-compose.yml for local orchestration.

a. Main Components:
- External Data Source: randomuser.me API
- Orchestration: Apache Airflow (Scheduler + Webserver + Worker)
- Relational Store: PostgreSQL (intermediate landing)
- Streaming Layer: Apache Kafka + Zookeeper
- Management: Confluent Control Center + Schema Registry
- Compute: Apache Spark cluster (Master + Workers running spark_stream.py)
- Final Storage: Apache Cassandra
- Docker Network: overarching container network
b. Relationships & Data Flow:
1. Airflow DAG → HTTP fetch → writes raw JSON to Postgres
2. Airflow (or a Kafka producer task) → reads from Postgres → publishes to Kafka topic
3. Zookeeper manages Kafka brokers; Schema Registry validates Avro/JSON schemas
4. Spark Streaming job (spark_stream.py) → consumes Kafka topic → transforms data
5. Spark → writes processed records to Cassandra
c. Architectural Patterns & Principles:
- Orchestration pattern (Airflow DAGs)
- Lambda-style streaming (micro-batch via Spark Streaming)
- Event-driven architecture (Kafka topics as event bus)
- Containerized microservices via Docker Compose
d. Key Technologies/Frameworks:
- Python (DAGs, producers)
- Apache Airflow for orchestration
- PostgreSQL as staging store
- Apache Kafka & Zookeeper for durable streaming
- Confluent Control Center & Schema Registry for monitoring/schemas
- Apache Spark for distributed processing
- Cassandra for NoSQL storage
- Docker Compose for orchestration of all services

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Source
    DataSource["randomuser.me API"]:::external

    subgraph "Docker Network"
        direction TB

        %% Orchestration Layer
        subgraph "Orchestration" 
            direction TB
            DockerCompose["docker-compose.yml"]:::config
            AirflowScheduler["Airflow Scheduler"]:::orchestration
            AirflowWebserver["Airflow Webserver"]:::orchestration
            AirflowWorker["Airflow Worker"]:::orchestration
            Postgres["PostgreSQL"]:::storage
            KafkaDAG["kafka_stream.py"]:::dag
            Entrypoint["entrypoint.sh"]:::script
        end

        %% Streaming Infrastructure
        subgraph "Streaming Infrastructure"
            direction TB
            Zookeeper["Zookeeper"]:::streaming
            KafkaBroker["Kafka Broker"]:::streaming
        end

        %% Kafka Ecosystem Tools
        subgraph "Kafka Ecosystem"
            direction TB
            SchemaRegistry["Schema Registry"]:::streaming
            ControlCenter["Control Center"]:::streaming
        end

        %% Processing Layer
        subgraph "Spark Cluster"
            direction TB
            SparkMaster["Spark Master"]:::processing
            SparkWorker["Spark Worker(s)"]:::processing
            SparkJob["spark_stream.py"]:::script
        end

        %% Final Storage
        subgraph "Storage Layer"
            direction TB
            Cassandra["Cassandra"]:::storage
        end
    end

    %% Data Flows
    DataSource -->|"HTTP Fetch"| AirflowScheduler
    AirflowScheduler -->|"runs kafka_stream.py"| KafkaDAG
    KafkaDAG -->|"writes raw JSON"| Postgres
    KafkaDAG -->|"publishes to topic"| KafkaBroker
    Postgres -->|"read records"| KafkaBroker

    Zookeeper -->|"manages"| KafkaBroker
    SchemaRegistry -->|"validates schemas"| KafkaBroker
    ControlCenter -->|"monitors"| KafkaBroker

    KafkaBroker -->|"streams data"| SparkMaster
    SparkMaster -->|"schedules jobs"| SparkWorker
    SparkWorker -->|"runs spark_stream.py"| SparkJob
    SparkJob -->|"writes processed records"| Cassandra

    Entrypoint -->|"initializes services"| Zookeeper
    Entrypoint -->|"initializes services"| AirflowScheduler

    %% Click Events
    click DockerCompose "https://github.com/airscholar/e2e-data-engineering/blob/main/docker-compose.yml"
    click KafkaDAG "https://github.com/airscholar/e2e-data-engineering/blob/main/dags/kafka_stream.py"
    click SparkJob "https://github.com/airscholar/e2e-data-engineering/blob/main/spark_stream.py"
    click Entrypoint "https://github.com/airscholar/e2e-data-engineering/blob/main/script/entrypoint.sh"
    click Requirements "https://github.com/airscholar/e2e-data-engineering/blob/main/requirements.txt"
    click Readme "https://github.com/airscholar/e2e-data-engineering/blob/main/README.md"
    click Gitignore "https://github.com/airscholar/e2e-data-engineering/blob/main/.gitignore"

    %% Misc Files
    Requirements["requirements.txt"]:::config
    Readme["README.md"]:::config
    Gitignore[".gitignore"]:::config

    %% Styles
    classDef orchestration fill:#cfe2f3,stroke:#3182bd,color:#000
    classDef streaming fill:#d4edda,stroke:#28a745,color:#000
    classDef processing fill:#ffe5b4,stroke:#fd7e14,color:#000
    classDef storage fill:#e2e3f3,stroke:#6f42c1,color:#000
    classDef config fill:#e2e3e5,stroke:#6c757d,color:#000
    classDef external fill:#f8f9fa,stroke:#6c757d,color:#000
    class KafkaDAG script
    class SparkJob script
    class Entrypoint script
    class Requirements config
    class Readme config
    class Gitignore config
```
