# eczachly/data-engineering-zoomcamp architecture

> Architecture diagram of the GitHub repository eczachly/data-engineering-zoomcamp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eczachly/data-engineering-zoomcamp
- Repository: https://github.com/eczachly/data-engineering-zoomcamp

## Overview

##### Project type and purpose
- This is not a single monolithic application but a learning‐platform repository (“Data Engineering Zoomcamp”) that delivers a sequence of modules and workshops.
- Its purpose is to teach end-to-end data engineering: containerization & infra as code, workflow orchestration, data ingestion, data warehousing, analytics engineering, batch processing, streaming, and a final capstone project.

##### File structure
- Top-level module folders:
- 01-docker-terraform (Docker, Terraform, GCP, PostgreSQL)
- 02-workflow-orchestration (Kestra, Docker Compose, Postgres)
- 03-data-warehouse (BigQuery SQL, ML, partitioning)
- 04-analytics-engineering (dbt, Docker, PostgreSQL & BigQuery)
- 05-batch (Apache Spark, PySpark notebooks, Hadoop/cluster configs)
- 06-streaming (Java Kafka examples, ksqlDB, Python Kafka/Redpanda, PyFlink)
- projects (final project templates)
- Cohorts folders replicate these modules in workshop/homework form per year.
- Configuration/deployment: Dockerfiles, docker-compose.yml, Terraform .tf, Airflow/Kestra .yaml.

##### Key architecture from README
- A slide (arch_v4_workshops.jpg) hints at the canonical end-to-end pipeline: ingestion → staging → orchestration → data warehouse → analytics → batch/streaming processing → serving/visualization.

a. Data Sources & Ingestion
- APIs, CSV/Parquet files, Pub/Sub/Kafka topics
- Ingestion scripts (Python, notebooks), Faust, custom producers/consumers
b. Container & Infra Layer
- Docker (images for PostgreSQL, Kafka, Spark, Airflow, Kestra)
- Terraform provisioning on GCP (compute, storage buckets, BigQuery datasets)
c. Workflow Orchestration
- Kestra + Docker Compose
- Alternative: Airflow DAGs in cohorts examples
d. Raw/Staging Storage
- GCS buckets or local volumes
- PostgreSQL Docker volumes (for analytics engineering)
e. Data Warehouse
- BigQuery (datasets, partitioned/clustering tables, ML)
- Postgres (for dbt dev)
f. Transformation & Analytics Engineering
- dbt Core/Cloud models, seeds, macros, tests
- Metabase integration (optional)
g. Batch Processing
- Spark cluster (master/workers via Docker or Kubernetes)
- Pyspark jobs, SQL notebooks, DataFrame APIs
h. Streaming Processing
- Kafka broker, Schema Registry (Avro), ksqlDB
- Kafka Streams (Java), Faust/PyFlink/PySpark Streaming
i. Presentation & Serving
- Final project code; dashboards via Metabase or other BI
- Slack/Github for community & code distribution

##### Relationships & Data Flows
- Ingestion → Raw Storage
- Orchestration → triggers ingestion, batch & streaming jobs
- Raw Storage → ETL/ELT (dbt, Spark, ksqlDB) → Data Warehouse
- Data Warehouse → downstream analytics/ML → serving/dashboards
- Streaming topics → real-time consumers → materialized views or DW

##### Architectural Patterns & Design Principles
- Modular, layered architecture (separation of concerns: ingest, store, process, serve)
- Infrastructure as Code (Terraform), Immutable containers (Docker)
- Declarative DAGs (Kestra/Airflow)
- Code-based transformations (dbt, Spark jobs)
- Event-driven streaming (Kafka + consumer apps)
- Polyglot approach (Python, Java, SQL, Avro schemas)

##### Final advice
- Both the “learning path” (module progression) and the “reference architecture” (end-to-end pipeline)
- Use your README’s arch_v4_workshops.jpg as a starting template; refine with tool names, ports, graph details
- With file paths or sample filenames where relevant for quick navigation back to code

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Ingestion Layer
    subgraph "Ingestion Layer"
        API["APIs"]
        CSV["CSV/Parquet Files"]
        KafkaTopic["Pub/Sub/Kafka Topics"]:::streaming
        ingest1["ingest_data.py"]:::compute
        ingest2["pipeline.py"]:::compute
        ingest3["data-loading-parquet.py"]:::compute
        ingest4["web_to_gcs.py"]:::compute
        ingest5["data_ingestion_gcs_dag.py"]:::compute
    end

    %% Infra Layer
    subgraph "Container & Infra Layer"
        Terraform["Terraform IaC (GCP)"]:::compute
        DockerfileSQL["Dockerfile (SQL)"]:::compute
        ComposeSQL["docker-compose.yaml (SQL)"]:::compute
        ComposeOrchInfra["docker-compose.yml (Orch Infra)"]:::compute
    end

    %% Orchestration Layer
    subgraph "Workflow Orchestration"
        KestraSvc["Kestra Services"]:::orchestration
        KestraFlows["Kestra DAGs"]:::orchestration
        AirflowDags["Airflow DAGs"]:::orchestration
        AirflowBQ["gcs_to_bq_dag.py"]:::orchestration
    end

    %% Storage Layer
    subgraph "Raw/Staging Storage"
        GCS["GCS Bucket"]:::storage
        PGVol["Postgres Volume"]:::storage
    end

    %% Data Warehouse Layer
    subgraph "Data Warehouse"
        BigQuery["BigQuery"]:::storage
        PostgresDW["Postgres (dbt dev)"]:::storage
    end

    %% Transformation Layer
    subgraph "Analytics Engineering"
        dbtDocker["Dockerfile (dbt)"]:::compute
        dbtCompose["docker-compose.yaml (dbt)"]:::compute
        dbtProj["dbt_project.yml"]:::compute
        dbtModels["dbt models"]:::compute
        dbtMacros["dbt macros"]:::compute
        dbtSeeds["taxi_zone_lookup.csv"]:::compute
    end

    %% Batch Processing Layer
    subgraph "Batch Processing"
        SparkCode["PySpark Notebooks & Scripts"]:::compute
        SparkDFConf["spark.dockerfile"]:::compute
        CoreSite["core-site.xml"]:::compute
        SparkDefaults["spark-defaults.conf"]:::compute
    end

    %% Streaming Processing Layer
    subgraph "Streaming Processing"
        KafkaBroker["Kafka Broker"]:::streaming
        SchemaReg["Schema Registry"]:::streaming
        ksqlDB["ksqlDB"]:::streaming
        JavaStreams["Java Kafka Streams"]:::streaming
        PyAvro["Python Avro Example"]:::streaming
        JSONExample["Python JSON Example"]:::streaming
        FaustApp["Faust Streaming App"]:::streaming
        PySparkStr["PySpark Streaming"]:::streaming
        KafkaCompose["docker-compose.yml (Kafka Broker)"]:::streaming
        SparkStreamCompose["docker-compose.yml (Spark Streaming)"]:::streaming
    end

    %% Serving Layer
    subgraph "Presentation & Serving"
        Dashboards["Metabase Dashboards"]:::serving
        FinalProj["Final Project Templates"]:::serving
        Datasets["Example Datasets"]:::serving
    end

    %% Data Flows
    API --> ingest1
    CSV --> ingest2
    KafkaTopic --> ingest3
    ingest1 --> GCS
    ingest2 --> GCS
    ingest3 --> GCS
    ingest4 --> GCS
    ingest5 --> GCS

    Terraform --> DockerfileSQL
    Terraform --> ComposeSQL
    Terraform --> ComposeOrchInfra
    DockerfileSQL --> DockerfileSQL
    ComposeSQL --> PGVol
    ComposeOrchInfra --> KestraSvc

    KestraSvc --> KestraFlows
    KestraSvc --> AirflowDags
    AirflowDags --> ingest5
    AirflowBQ --> BigQuery

    GCS --> PostgresDW
    GCS --> BigQuery

    PostgresDW --> dbtProj
    BigQuery --> dbtProj

    dbtProj --> dbtModels
    dbtProj --> dbtMacros
    dbtProj --> dbtSeeds

    GCS --> SparkCode
    SparkCode --> SparkDFConf
    SparkCode --> CoreSite
    SparkCode --> SparkDefaults
    SparkCode --> PostgresDW

    KafkaBroker --> SchemaReg
    KafkaBroker --> ksqlDB
    SchemaReg --> JavaStreams
    SchemaReg --> PyAvro
    SchemaReg --> JSONExample
    JavaStreams --> BigQuery
    PyAvro --> BigQuery
    JSONExample --> BigQuery
    FaustApp --> BigQuery
    PySparkStr --> BigQuery
    SparkStreamCompose --> PySparkStr

    BigQuery --> Dashboards
    PostgresDW --> Dashboards
    Dashboards --> FinalProj
    Dashboards --> Datasets

    %% Click Events - Ingestion
    click ingest1 "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/01-docker-terraform/2_docker_sql/ingest_data.py"
    click ingest2 "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/01-docker-terraform/2_docker_sql/pipeline.py"
    click ingest3 "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/01-docker-terraform/2_docker_sql/data-loading-parquet.py"
    click ingest4 "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/03-data-warehouse/extras/web_to_gcs.py"
    click ingest5 "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/cohorts/2022/week_2_data_ingestion/airflow/dags/data_ingestion_gcs_dag.py"

    %% Click Events - Infra
    click Terraform "https://github.com/eczachly/data-engineering-zoomcamp/tree/main/01-docker-terraform/1_terraform_gcp/terraform"
    click DockerfileSQL "https://github.com/eczachly/data-engineering-zoomcamp/tree/main/01-docker-terraform/2_docker_sql/Dockerfile"
    click ComposeSQL "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/01-docker-terraform/2_docker_sql/docker-compose.yaml"
    click ComposeOrchInfra "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/02-workflow-orchestration/docker/combined/docker-compose.yml"

    %% Click Events - Orchestration
    click KestraSvc "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/02-workflow-orchestration/docker/kestra/docker-compose.yml"
    click KestraFlows "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/02-workflow-orchestration/flows/*.yaml"
    click AirflowDags "https://github.com/eczachly/data-engineering-zoomcamp/tree/main/cohorts/2022/week_2_data_ingestion/airflow/dags"
    click AirflowBQ "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/cohorts/2022/week_3_data_warehouse/airflow/dags/gcs_to_bq_dag.py"

    %% Click Events - Data Warehouse
    click BigQuery "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/03-data-warehouse/big_query.sql"
    click BigQuery "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/03-data-warehouse/big_query_ml.sql"
    click BigQuery "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/03-data-warehouse/big_query_hw.sql"

    %% Click Events - Transformation
    click dbtDocker "https://github.com/eczachly/data-engineering-zoomcamp/tree/main/04-analytics-engineering/docker_setup/Dockerfile"
    click dbtCompose "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/04-analytics-engineering/docker_setup/docker-compose.yaml"
    click dbtProj "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/04-analytics-engineering/taxi_rides_ny/dbt_project.yml"
    click dbtModels "https://github.com/eczachly/data-engineering-zoomcamp/tree/main/04-analytics-engineering/taxi_rides_ny/models/"
    click dbtMacros "https://github.com/eczachly/data-engineering-zoomcamp/tree/main/04-analytics-engineering/taxi_rides_ny/macros/"
    click dbtSeeds "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/04-analytics-engineering/taxi_rides_ny/seeds/taxi_zone_lookup.csv"

    %% Click Events - Batch
    click SparkCode "https://github.com/eczachly/data-engineering-zoomcamp/tree/main/05-batch/code"
    click SparkDFConf "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/05-batch/setup/config/spark.dockerfile"
    click CoreSite "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/05-batch/setup/config/core-site.xml"
    click SparkDefaults "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/05-batch/setup/config/spark-defaults.conf"

    %% Click Events - Streaming
    click JavaStreams "https://github.com/eczachly/data-engineering-zoomcamp/tree/main/06-streaming/java/kafka_examples"
    click PyAvro "https://github.com/eczachly/data-engineering-zoomcamp/tree/main/06-streaming/python/avro_example"
    click JSONExample "https://github.com/eczachly/data-engineering-zoomcamp/tree/main/06-streaming/python/json_example"
    click FaustApp "https://github.com/eczachly/data-engineering-zoomcamp/tree/main/06-streaming/python/streams-example/faust"
    click PySparkStr "https://github.com/eczachly/data-engineering-zoomcamp/tree/main/06-streaming/python/streams-example/pyspark"
    click KafkaCompose "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/06-streaming/python/docker/kafka/docker-compose.yml"
    click SparkStreamCompose "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/06-streaming/python/docker/spark/docker-compose.yml"

    %% Click Events - Serving
    click FinalProj "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/projects/README.md"
    click Datasets "https://github.com/eczachly/data-engineering-zoomcamp/blob/main/projects/datasets.md"

    %% Styles
    classDef compute fill:#D6EAF8,stroke:#3498DB,color:#1B4F72
    classDef storage fill:#D5F5E3,stroke:#27AE60,color:#145A32
    classDef orchestration fill:#FDEDEC,stroke:#E74C3C,color:#641E16
    classDef streaming fill:#E8DAEF,stroke:#8E44AD,color:#4A235A
    classDef serving fill:#FCF3CF,stroke:#F1C40F,color:#7D6608
```
