How eczachly/data-engineering-zoomcamp works
Project type and purpose
• This is not a single monolithic application but a learning‐platform repository (“Data Engineering Zoomcamp”) that delivers a sequence of modules and workshops.
• Its purpose is to teach end-to-end data engineering: containerization & infra as code, workflow orchestration, data ingestion, data warehousing, analytics engineering, batch processing, streaming, and a final capstone project.
File structure
• Top-level module folders:
• 01-docker-terraform (Docker, Terraform, GCP, PostgreSQL)
• 02-workflow-orchestration (Kestra, Docker Compose, Postgres)
• 03-data-warehouse (BigQuery SQL, ML, partitioning)
• 04-analytics-engineering (dbt, Docker, PostgreSQL & BigQuery)
• 05-batch (Apache Spark, PySpark notebooks, Hadoop/cluster configs)
• 06-streaming (Java Kafka examples, ksqlDB, Python Kafka/Redpanda, PyFlink)
• projects (final project templates)
• Cohorts folders replicate these modules in workshop/homework form per year.
• Configuration/deployment: Dockerfiles, docker-compose.yml, Terraform .tf, Airflow/Kestra .yaml.
Key architecture from README
• A slide (arch_v4_workshops.jpg) hints at the canonical end-to-end pipeline: ingestion → staging → orchestration → data warehouse → analytics → batch/streaming processing → serving/visualization.
a. Data Sources & Ingestion
• APIs, CSV/Parquet files, Pub/Sub/Kafka topics
• Ingestion scripts (Python, notebooks), Faust, custom producers/consumers
b. Container & Infra Layer
• Docker (images for PostgreSQL, Kafka, Spark, Airflow, Kestra)
• Terraform provisioning on GCP (compute, storage buckets, BigQuery datasets)
c. Workflow Orchestration
• Kestra + Docker Compose
• Alternative: Airflow DAGs in cohorts examples
d. Raw/Staging Storage
• GCS buckets or local volumes
• PostgreSQL Docker volumes (for analytics engineering)
e. Data Warehouse
• BigQuery (datasets, partitioned/clustering tables, ML)
• Postgres (for dbt dev)
f. Transformation & Analytics Engineering
• dbt Core/Cloud models, seeds, macros, tests
• Metabase integration (optional)
g. Batch Processing
• Spark cluster (master/workers via Docker or Kubernetes)
• Pyspark jobs, SQL notebooks, DataFrame APIs
h. Streaming Processing
• Kafka broker, Schema Registry (Avro), ksqlDB
• Kafka Streams (Java), Faust/PyFlink/PySpark Streaming
i. Presentation & Serving
• Final project code; dashboards via Metabase or other BI
• Slack/Github for community & code distribution
Relationships & Data Flows
• Ingestion → Raw Storage
• Orchestration → triggers ingestion, batch & streaming jobs
• Raw Storage → ETL/ELT (dbt, Spark, ksqlDB) → Data Warehouse
• Data Warehouse → downstream analytics/ML → serving/dashboards
• Streaming topics → real-time consumers → materialized views or DW
Architectural Patterns & Design Principles
• Modular, layered architecture (separation of concerns: ingest, store, process, serve)
• Infrastructure as Code (Terraform), Immutable containers (Docker)
• Declarative DAGs (Kestra/Airflow)
• Code-based transformations (dbt, Spark jobs)
• Event-driven streaming (Kafka + consumer apps)
• Polyglot approach (Python, Java, SQL, Avro schemas)
Final advice
• Both the “learning path” (module progression) and the “reference architecture” (end-to-end pipeline)
• Use your README’s arch_v4_workshops.jpg as a starting template; refine with tool names, ports, graph details
• With file paths or sample filenames where relevant for quick navigation back to code