How datatalksclub/data-engineering-zoomcamp works
This repository is a free data-engineering course, not one deployed pipeline. Its user journey is from course materials through hands-on ingestion, warehousing, batch, and streaming exercises to a final project. The graph highlights the runnable examples visible in the supplied tree and sampled source; README-described topics without a source path in that tree are not mapped as implementation components.
Main entry points: consumer_postgres.py Postgres consumer · producer.py Avro producer · data_ingestion_local.py Local Airflow DAG · producer_realtime.py Realtime producer
Components 20
Course journey
- Course materials
- Final project
Data ingestion
- Local Airflow DAG
cohorts/2022/week_2_data_ingestion/airflow/dags_local/data_ingestion_local.py - Chunked CSV ingest
cohorts/2022/week_2_data_ingestion/airflow/dags_local/ingest_script.py - PostgreSQL
- GCS ingestion
cohorts/2022/week_2_data_ingestion/airflow/dags/data_ingestion_gcs_dag.py
Warehousing and batch
- Taxi data loading
03-data-warehouse/load_yellow_taxi_data.py - Spark SQL
06-batch/code/06_spark_sql.py - Spark to BigQuery
06-batch/code/06_spark_sql_big_query.py
Streaming examples
- Ride model
07-streaming/code/src/models.py - Realtime producer
07-streaming/code/src/producers/producer_realtime.py - Kafka
- Postgres consumer
07-streaming/code/src/consumers/consumer_postgres.py - Spark stream processing
cohorts/2023/week_6_stream_processing/streaming_confluent.py - Taxi CSV producer
cohorts/2023/week_6_stream_processing/producer_confluent.py - Avro producer
07-streaming/extras/extras/python/avro_example/producer.py - Avro ride key
07-streaming/extras/extras/python/avro_example/ride_record_key.py - Avro ride record
07-streaming/extras/extras/python/avro_example/ride_record.py - Schema Registry
Other components
- Learner
Connections 15
- Learner → Course materials: follows
- Course materials → Final project: applies learning
- Local Airflow DAG → Chunked CSV ingest: calls ingest
cohorts/2022/week_2_data_ingestion/airflow/dags_local/data_ingestion_local.py - Chunked CSV ingest → PostgreSQL: writes rows
- Local Airflow DAG → PostgreSQL: configures connection
cohorts/2022/week_2_data_ingestion/airflow/dags_local/data_ingestion_local.py - Realtime producer → Ride model: creates rides
07-streaming/code/src/producers/producer_realtime.py - Realtime producer → Kafka: publishes rides
07-streaming/code/src/producers/producer_realtime.py - Postgres consumer → Ride model: deserializes rides
07-streaming/code/src/consumers/consumer_postgres.py - Postgres consumer → Kafka: consumes rides
07-streaming/code/src/consumers/consumer_postgres.py - Postgres consumer → PostgreSQL: inserts events
07-streaming/code/src/consumers/consumer_postgres.py - Taxi CSV producer → Kafka: publishes records
cohorts/2023/week_6_stream_processing/producer_confluent.py - Spark stream processing → Kafka: reads and writes
cohorts/2023/week_6_stream_processing/streaming_confluent.py - Avro producer → Avro ride key: serializes key
07-streaming/extras/extras/python/avro_example/producer.py - Avro producer → Avro ride record: serializes value
07-streaming/extras/extras/python/avro_example/producer.py - Avro producer → Schema Registry: registers schemas
07-streaming/extras/extras/python/avro_example/producer.py