# datatalksclub/data-engineering-zoomcamp architecture

> Architecture diagram of the GitHub repository datatalksclub/data-engineering-zoomcamp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datatalksclub/data-engineering-zoomcamp
- Repository: https://github.com/datatalksclub/data-engineering-zoomcamp

## Overview

This repository is a free data-engineering course, not one deployed pipeline. Its user journey is from course materials through hands-on ingestion, warehousing, batch, and streaming exercises to a final project. The graph highlights the runnable examples visible in the supplied tree and sampled source; README-described topics without a source path in that tree are not mapped as implementation components.

Main entry points: [`07-streaming/code/src/consumers/consumer_postgres.py`](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/code/src/consumers/consumer_postgres.py) (Postgres consumer), [`07-streaming/extras/extras/python/avro_example/producer.py`](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/extras/extras/python/avro_example/producer.py) (Avro producer), [`cohorts/2022/week_2_data_ingestion/airflow/dags_local/data_ingestion_local.py`](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/cohorts/2022/week_2_data_ingestion/airflow/dags_local/data_ingestion_local.py) (Local Airflow DAG), [`07-streaming/code/src/producers/producer_realtime.py`](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/code/src/producers/producer_realtime.py) (Realtime producer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_learning["Course journey"]
  node_materials["Course materials"]
  node_final["Final project"]
end

subgraph group_ingestion["Data ingestion"]
  node_localdag["Local Airflow DAG"]
  node_ingestscript["Chunked CSV ingest<br/>[ingest_script.py]"]
  node_postgres[("PostgreSQL")]
  node_gcsingest["GCS ingestion"]
end

subgraph group_warehouse["Warehousing and batch"]
  node_warehouse_load["Taxi data loading"]
  node_spark_sql["Spark SQL<br/>[06_spark_sql.py]"]
  node_spark_bq["Spark to BigQuery"]
end

subgraph group_streaming["Streaming examples"]
  node_ride_model["Ride model<br/>[models.py]"]
  node_realtime_producer["Realtime producer"]
  node_kafka["Kafka"]
  node_postgres_consumer["Postgres consumer"]
  node_spark_stream["Spark stream processing"]
  node_confluent_producer["Taxi CSV producer"]
  node_avro_producer["Avro producer<br/>[producer.py]"]
  node_avro_key["Avro ride key<br/>[ride_record_key.py]"]
  node_avro_record["Avro ride record<br/>[ride_record.py]"]
  node_schema_registry["Schema Registry"]
end

node_learner(("Learner"))

node_learner -->|"follows"| node_materials
node_materials -->|"applies learning"| node_final
node_localdag -->|"calls ingest"| node_ingestscript
node_ingestscript -.->|"writes rows"| node_postgres
node_localdag -->|"configures connection"| node_postgres
node_realtime_producer -->|"creates rides"| node_ride_model
node_realtime_producer -->|"publishes rides"| node_kafka
node_postgres_consumer -->|"deserializes rides"| node_ride_model
node_postgres_consumer -->|"consumes rides"| node_kafka
node_postgres_consumer -->|"inserts events"| node_postgres
node_confluent_producer -->|"publishes records"| node_kafka
node_spark_stream -->|"reads and writes"| node_kafka
node_avro_producer -->|"serializes key"| node_avro_key
node_avro_producer -->|"serializes value"| node_avro_record
node_avro_producer -->|"registers schemas"| node_schema_registry

click node_localdag "https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/cohorts/2022/week_2_data_ingestion/airflow/dags_local/data_ingestion_local.py"
click node_ingestscript "https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/cohorts/2022/week_2_data_ingestion/airflow/dags_local/ingest_script.py"
click node_gcsingest "https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/cohorts/2022/week_2_data_ingestion/airflow/dags/data_ingestion_gcs_dag.py"
click node_warehouse_load "https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/03-data-warehouse/load_yellow_taxi_data.py"
click node_spark_sql "https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/06-batch/code/06_spark_sql.py"
click node_spark_bq "https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/06-batch/code/06_spark_sql_big_query.py"
click node_ride_model "https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/code/src/models.py"
click node_realtime_producer "https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/code/src/producers/producer_realtime.py"
click node_postgres_consumer "https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/code/src/consumers/consumer_postgres.py"
click node_spark_stream "https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/cohorts/2023/week_6_stream_processing/streaming_confluent.py"
click node_confluent_producer "https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/cohorts/2023/week_6_stream_processing/producer_confluent.py"
click node_avro_producer "https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/extras/extras/python/avro_example/producer.py"
click node_avro_key "https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/extras/extras/python/avro_example/ride_record_key.py"
click node_avro_record "https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/extras/extras/python/avro_example/ride_record.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_materials,node_final toneBlue
class node_localdag,node_ingestscript,node_postgres,node_gcsingest toneAmber
class node_warehouse_load,node_spark_sql,node_spark_bq toneMint
class node_ride_model,node_realtime_producer,node_kafka,node_postgres_consumer,node_spark_stream,node_confluent_producer,node_avro_producer,node_avro_key,node_avro_record,node_schema_registry toneRose
class node_learner toneIndigo
```

## Components

### Course journey

- **Course materials**
- **Final project**

### Data ingestion

- [Local Airflow DAG](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/cohorts/2022/week_2_data_ingestion/airflow/dags_local/data_ingestion_local.py): `cohorts/2022/week_2_data_ingestion/airflow/dags_local/data_ingestion_local.py`
- [Chunked CSV ingest](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/cohorts/2022/week_2_data_ingestion/airflow/dags_local/ingest_script.py): `cohorts/2022/week_2_data_ingestion/airflow/dags_local/ingest_script.py`
- **PostgreSQL**
- [GCS ingestion](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/cohorts/2022/week_2_data_ingestion/airflow/dags/data_ingestion_gcs_dag.py): `cohorts/2022/week_2_data_ingestion/airflow/dags/data_ingestion_gcs_dag.py`

### Warehousing and batch

- [Taxi data loading](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/03-data-warehouse/load_yellow_taxi_data.py): `03-data-warehouse/load_yellow_taxi_data.py`
- [Spark SQL](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/06-batch/code/06_spark_sql.py): `06-batch/code/06_spark_sql.py`
- [Spark to BigQuery](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/06-batch/code/06_spark_sql_big_query.py): `06-batch/code/06_spark_sql_big_query.py`

### Streaming examples

- [Ride model](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/code/src/models.py): `07-streaming/code/src/models.py`
- [Realtime producer](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/code/src/producers/producer_realtime.py): `07-streaming/code/src/producers/producer_realtime.py`
- **Kafka**
- [Postgres consumer](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/code/src/consumers/consumer_postgres.py): `07-streaming/code/src/consumers/consumer_postgres.py`
- [Spark stream processing](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/cohorts/2023/week_6_stream_processing/streaming_confluent.py): `cohorts/2023/week_6_stream_processing/streaming_confluent.py`
- [Taxi CSV producer](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/cohorts/2023/week_6_stream_processing/producer_confluent.py): `cohorts/2023/week_6_stream_processing/producer_confluent.py`
- [Avro producer](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/extras/extras/python/avro_example/producer.py): `07-streaming/extras/extras/python/avro_example/producer.py`
- [Avro ride key](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/extras/extras/python/avro_example/ride_record_key.py): `07-streaming/extras/extras/python/avro_example/ride_record_key.py`
- [Avro ride record](https://github.com/datatalksclub/data-engineering-zoomcamp/blob/main/07-streaming/extras/extras/python/avro_example/ride_record.py): `07-streaming/extras/extras/python/avro_example/ride_record.py`
- **Schema Registry**

### Other components

- **Learner**

## Connections

- Learner → Course materials: follows
- Course materials → Final project: applies learning
- Local Airflow DAG → Chunked CSV ingest: calls ingest (evidence: `cohorts/2022/week_2_data_ingestion/airflow/dags_local/data_ingestion_local.py`)
- Chunked CSV ingest → PostgreSQL: writes rows
- Local Airflow DAG → PostgreSQL: configures connection (evidence: `cohorts/2022/week_2_data_ingestion/airflow/dags_local/data_ingestion_local.py`)
- Realtime producer → Ride model: creates rides (evidence: `07-streaming/code/src/producers/producer_realtime.py`)
- Realtime producer → Kafka: publishes rides (evidence: `07-streaming/code/src/producers/producer_realtime.py`)
- Postgres consumer → Ride model: deserializes rides (evidence: `07-streaming/code/src/consumers/consumer_postgres.py`)
- Postgres consumer → Kafka: consumes rides (evidence: `07-streaming/code/src/consumers/consumer_postgres.py`)
- Postgres consumer → PostgreSQL: inserts events (evidence: `07-streaming/code/src/consumers/consumer_postgres.py`)
- Taxi CSV producer → Kafka: publishes records (evidence: `cohorts/2023/week_6_stream_processing/producer_confluent.py`)
- Spark stream processing → Kafka: reads and writes (evidence: `cohorts/2023/week_6_stream_processing/streaming_confluent.py`)
- Avro producer → Avro ride key: serializes key (evidence: `07-streaming/extras/extras/python/avro_example/producer.py`)
- Avro producer → Avro ride record: serializes value (evidence: `07-streaming/extras/extras/python/avro_example/producer.py`)
- Avro producer → Schema Registry: registers schemas (evidence: `07-streaming/extras/extras/python/avro_example/producer.py`)
