How alanchn31/data-engineering-projects works
Overall project type and purpose
• This repo is a collection of stand-alone data engineering projects demonstrating end-to-end ETL, data modelling, warehousing, lake formation, web scraping and orchestration patterns.
• Each numbered top-level folder (e.g., “1. Postgres ETL”, “2. Cassandra ETL”, “3. Web Scraping…”, etc.) is its own pipeline, ingesting data from a source, transforming it, and loading it into a target store.
File structure for architectural clues
• Top‐level folders (“1. Postgres ETL”, “2. Cassandra ETL”, etc.) map to discrete pipelines.
• raw data (JSON, CSV, website, S3)
• ingestion/transformation code (Python scripts, Jupyter notebooks, Spark jobs, Scrapy spiders)
• schema/configuration files (sql_queries.py, create_tables.py, dwh.cfg)
• The “0. Back to Basics” folder is documentation/exercises; “6. Data Pipelining with Airflow” is an orchestration layer that ties some pipelines together.
README for additional insights
• Describes six completed projects and a capstone.
• Technologies: Postgres, Cassandra, MongoDB, AWS Redshift, AWS S3, Spark, Scrapy, Airflow.
• Each project answers business questions about a music startup or a books website.
Main components
a. Data Sources
– JSON log_data and song_data files (local FS)
– CSV event_data files for Cassandra
– Books website (HTTP endpoint)
– AWS S3 buckets containing JSON for the data lake
b. ETL/Processing Modules
– Python scripts & Jupyter notebooks (Postgres ETL, Cassandra ETL, Redshift ETL, Data Lake)
– Scrapy spider + pipelines (Web Scraping → MongoDB)
– Spark application (Spark & S3 Data Lake)
– Airflow DAGs & custom operators/plugins (orchestrates S3→Redshift loads)
c. Storage Targets
– PostgreSQL OLAP DB (star schema)
– Cassandra NoSQL cluster (query-centric tables)
– MongoDB collection (books documents)
– AWS Redshift data warehouse
– AWS S3 data lake (Parquet files)
d. Orchestration Layer
– Apache Airflow (DAGs, custom operators for staging, fact/dimension loads, data quality checks)
Relationships and data flows
• For the Data Lake + Redshift projects, show the two-step pipeline: JSON in S3 → Spark job → Parquet in S3 → Airflow DAG → Redshift staging → Redshift fact/dimension tables.
• That Airflow DAGs import SQL scripts and call Python functions to create tables and run ETL.
Architectural patterns & design principles
• Batch ETL (daily JSON/CSV loads)
• Schema-on-write vs schema-on-read (Postgres star schema vs Spark/S3)
• Query-centric NoSQL modelling (Cassandra)
• Document store for semi-structured data (MongoDB)
• Data lake + data warehouse coexistence
• Workflow-as-code (Airflow DAGs & custom operators)
Technologies, frameworks & libraries
• Python (etl.py, create_tables.py)
• Jupyter Notebooks (exploratory ETL, tutorials)
• psycopg2 / sql_queries.py (Postgres)
• cassandra-driver (Cassandra notebooks)
• Scrapy (items, pipelines, spiders)
• PyMongo (MongoDB pipeline)
• boto3 & Spark (PySpark)
• airflow (DAGs, plugins/operators)