# selimunal/data-engineering-zoomcamp architecture

> Architecture diagram of the GitHub repository selimunal/data-engineering-zoomcamp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/selimunal/data-engineering-zoomcamp
- Repository: https://github.com/selimunal/data-engineering-zoomcamp

## Overview

Data Engineering Zoomcamp is a self-paced and cohort-based educational course, not a single data-processing service. Its main learner workflow is to follow the syllabus from foundations through orchestration, warehousing, analytics, batch and streaming, then apply the material in a project. The README also documents community support, a course UI and a workshop. Source examples illustrate selected lessons; they are not a unified runtime pipeline. The graph uses README statements and the supplied source excerpts/index for its connections; other lesson contents were not inspected.

Main entry points: [`README.md`](https://github.com/selimunal/data-engineering-zoomcamp/blob/main/README.md) (Course syllabus), [`week_6_stream_processing`](https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_6_stream_processing) (Stream processing), [`week_6_stream_processing/python/avro_example/producer.py`](https://github.com/selimunal/data-engineering-zoomcamp/blob/main/week_6_stream_processing/python/avro_example/producer.py) (Avro producer), [`week_2_workflow_orchestration`](https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_2_workflow_orchestration) (Orchestration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_course["Course curriculum"]
  node_syllabus["Course syllabus<br/>[README.md]"]
  node_foundations["Foundations"]
  node_orchestration["Orchestration"]
  node_warehouse["Data warehouse"]
  node_analytics["Analytics engineering"]
  node_batch["Batch processing"]
  node_streaming["Stream processing"]
end

subgraph group_examples["Lesson examples"]
  node_ingest_dag["Ingestion workflow"]
  node_ingest_loader["CSV loader<br/>[ingest_script.py]"]
  node_stream_demo["Spark stream demo"]
  node_stream_config["Stream settings<br/>[settings.py]"]
  node_avro_producer["Avro producer<br/>[producer.py]"]
  node_avro_record["Avro ride record<br/>[ride_record.py]"]
  node_avro_key["Avro record key<br/>[ride_record_key.py]"]
  node_json_ride["JSON ride model<br/>[ride.py]"]
  node_faust_ride["Faust ride model<br/>[taxi_rides.py]"]
end

subgraph group_community["Learner support"]
  node_community["Community support"]
  node_course_ui["Course UI"]
end

subgraph group_application["Course application"]
  node_project["Learner project"]
  node_workshop["dbt and PipeRider workshop<br/>[piperider.md]"]
  node_cohort["Cohort materials"]
end

node_learner(("Learner"))

node_learner -->|"follows"| node_syllabus
node_syllabus -->|"starts with"| node_foundations
node_syllabus -->|"continues to"| node_orchestration
node_syllabus -->|"continues to"| node_warehouse
node_syllabus -->|"continues to"| node_analytics
node_syllabus -->|"continues to"| node_batch
node_syllabus -->|"continues to"| node_streaming
node_orchestration -->|"teaches examples"| node_ingest_dag
node_ingest_dag -->|"calls"| node_ingest_loader
node_streaming -->|"teaches examples"| node_stream_demo
node_stream_demo -->|"imports settings"| node_stream_config
node_streaming -->|"teaches examples"| node_avro_producer
node_avro_producer -->|"serializes records"| node_avro_record
node_avro_producer -->|"serializes keys"| node_avro_key
node_streaming -->|"teaches examples"| node_json_ride
node_streaming -->|"teaches examples"| node_faust_ride
node_learner -->|"seeks help"| node_community
node_learner -->|"navigates course"| node_course_ui
node_learner -->|"joins cohort"| node_cohort
node_learner -->|"applies learning"| node_project
node_learner -->|"explores workshop"| node_workshop

click node_syllabus "https://github.com/selimunal/data-engineering-zoomcamp/blob/main/README.md"
click node_foundations "https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_1_basics_n_setup"
click node_orchestration "https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_2_workflow_orchestration"
click node_warehouse "https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_3_data_warehouse"
click node_analytics "https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_4_analytics_engineering"
click node_batch "https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_5_batch_processing"
click node_streaming "https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_6_stream_processing"
click node_ingest_dag "https://github.com/selimunal/data-engineering-zoomcamp/blob/main/cohorts/2022/week_2_data_ingestion/airflow/dags_local/data_ingestion_local.py"
click node_ingest_loader "https://github.com/selimunal/data-engineering-zoomcamp/blob/main/cohorts/2022/week_2_data_ingestion/airflow/dags_local/ingest_script.py"
click node_stream_demo "https://github.com/selimunal/data-engineering-zoomcamp/blob/main/cohorts/2023/week_6_stream_processing/streaming_confluent.py"
click node_stream_config "https://github.com/selimunal/data-engineering-zoomcamp/blob/main/cohorts/2023/week_6_stream_processing/settings.py"
click node_avro_producer "https://github.com/selimunal/data-engineering-zoomcamp/blob/main/week_6_stream_processing/python/avro_example/producer.py"
click node_avro_record "https://github.com/selimunal/data-engineering-zoomcamp/blob/main/week_6_stream_processing/python/avro_example/ride_record.py"
click node_avro_key "https://github.com/selimunal/data-engineering-zoomcamp/blob/main/week_6_stream_processing/python/avro_example/ride_record_key.py"
click node_json_ride "https://github.com/selimunal/data-engineering-zoomcamp/blob/main/week_6_stream_processing/python/json_example/ride.py"
click node_faust_ride "https://github.com/selimunal/data-engineering-zoomcamp/blob/main/week_6_stream_processing/python/streams-example/faust/taxi_rides.py"
click node_community "https://github.com/selimunal/data-engineering-zoomcamp/blob/main/asking-questions.md"
click node_project "https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_7_project"
click node_workshop "https://github.com/selimunal/data-engineering-zoomcamp/blob/main/cohorts/2023/workshops/piperider.md"
click node_cohort "https://github.com/selimunal/data-engineering-zoomcamp/tree/main/cohorts/2023"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_syllabus,node_foundations,node_orchestration,node_warehouse,node_analytics,node_batch,node_streaming toneBlue
class node_ingest_dag,node_ingest_loader,node_stream_demo,node_stream_config,node_avro_producer,node_avro_record,node_avro_key,node_json_ride,node_faust_ride toneAmber
class node_community,node_course_ui toneMint
class node_project,node_workshop,node_cohort toneRose
class node_learner toneIndigo
```

## Components

### Course curriculum

- [Course syllabus](https://github.com/selimunal/data-engineering-zoomcamp/blob/main/README.md): `README.md`
- [Foundations](https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_1_basics_n_setup): `week_1_basics_n_setup`
- [Orchestration](https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_2_workflow_orchestration): `week_2_workflow_orchestration`
- [Data warehouse](https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_3_data_warehouse): `week_3_data_warehouse`
- [Analytics engineering](https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_4_analytics_engineering): `week_4_analytics_engineering`
- [Batch processing](https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_5_batch_processing): `week_5_batch_processing`
- [Stream processing](https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_6_stream_processing): `week_6_stream_processing`

### Lesson examples

- [Ingestion workflow](https://github.com/selimunal/data-engineering-zoomcamp/blob/main/cohorts/2022/week_2_data_ingestion/airflow/dags_local/data_ingestion_local.py): `cohorts/2022/week_2_data_ingestion/airflow/dags_local/data_ingestion_local.py`
- [CSV loader](https://github.com/selimunal/data-engineering-zoomcamp/blob/main/cohorts/2022/week_2_data_ingestion/airflow/dags_local/ingest_script.py): `cohorts/2022/week_2_data_ingestion/airflow/dags_local/ingest_script.py`
- [Spark stream demo](https://github.com/selimunal/data-engineering-zoomcamp/blob/main/cohorts/2023/week_6_stream_processing/streaming_confluent.py): `cohorts/2023/week_6_stream_processing/streaming_confluent.py`
- [Stream settings](https://github.com/selimunal/data-engineering-zoomcamp/blob/main/cohorts/2023/week_6_stream_processing/settings.py): `cohorts/2023/week_6_stream_processing/settings.py`
- [Avro producer](https://github.com/selimunal/data-engineering-zoomcamp/blob/main/week_6_stream_processing/python/avro_example/producer.py): `week_6_stream_processing/python/avro_example/producer.py`
- [Avro ride record](https://github.com/selimunal/data-engineering-zoomcamp/blob/main/week_6_stream_processing/python/avro_example/ride_record.py): `week_6_stream_processing/python/avro_example/ride_record.py`
- [Avro record key](https://github.com/selimunal/data-engineering-zoomcamp/blob/main/week_6_stream_processing/python/avro_example/ride_record_key.py): `week_6_stream_processing/python/avro_example/ride_record_key.py`
- [JSON ride model](https://github.com/selimunal/data-engineering-zoomcamp/blob/main/week_6_stream_processing/python/json_example/ride.py): `week_6_stream_processing/python/json_example/ride.py`
- [Faust ride model](https://github.com/selimunal/data-engineering-zoomcamp/blob/main/week_6_stream_processing/python/streams-example/faust/taxi_rides.py): `week_6_stream_processing/python/streams-example/faust/taxi_rides.py`

### Learner support

- [Community support](https://github.com/selimunal/data-engineering-zoomcamp/blob/main/asking-questions.md): `asking-questions.md`
- **Course UI**

### Course application

- [Learner project](https://github.com/selimunal/data-engineering-zoomcamp/tree/main/week_7_project): `week_7_project`
- [dbt and PipeRider workshop](https://github.com/selimunal/data-engineering-zoomcamp/blob/main/cohorts/2023/workshops/piperider.md): `cohorts/2023/workshops/piperider.md`
- [Cohort materials](https://github.com/selimunal/data-engineering-zoomcamp/tree/main/cohorts/2023): `cohorts/2023`

### Other components

- **Learner**

## Connections

- Learner → Course syllabus: follows (evidence: `README.md`)
- Course syllabus → Foundations: starts with (evidence: `README.md`)
- Course syllabus → Orchestration: continues to (evidence: `README.md`)
- Course syllabus → Data warehouse: continues to (evidence: `README.md`)
- Course syllabus → Analytics engineering: continues to (evidence: `README.md`)
- Course syllabus → Batch processing: continues to (evidence: `README.md`)
- Course syllabus → Stream processing: continues to (evidence: `README.md`)
- Orchestration → Ingestion workflow: teaches examples (evidence: `README.md`)
- Ingestion workflow → CSV loader: calls (evidence: `cohorts/2022/week_2_data_ingestion/airflow/dags_local/data_ingestion_local.py`)
- Stream processing → Spark stream demo: teaches examples (evidence: `README.md`)
- Spark stream demo → Stream settings: imports settings (evidence: `cohorts/2023/week_6_stream_processing/streaming_confluent.py`)
- Stream processing → Avro producer: teaches examples (evidence: `README.md`)
- Avro producer → Avro ride record: serializes records (evidence: `week_6_stream_processing/python/avro_example/producer.py`)
- Avro producer → Avro record key: serializes keys (evidence: `week_6_stream_processing/python/avro_example/producer.py`)
- Stream processing → JSON ride model: teaches examples (evidence: `README.md`)
- Stream processing → Faust ride model: teaches examples (evidence: `README.md`)
- Learner → Community support: seeks help (evidence: `README.md`)
- Learner → Course UI: navigates course (evidence: `README.md`)
- Learner → Cohort materials: joins cohort (evidence: `README.md`)
- Learner → Learner project: applies learning (evidence: `README.md`)
- Learner → dbt and PipeRider workshop: explores workshop (evidence: `README.md`)
