How darshilparmar/datavidhya-data-engineering-course-resources works
This repository is a collection of independent data-engineering projects and examples, not one deployable application. Its workflows cover API extraction, orchestration, transformation, streaming, lakehouse storage, and serving. The graph highlights the sampled AirLens, Spotify, crypto, and City Bikes workflows; directory-level nodes represent other substantial project areas visible in the tree. The README is empty, and unsampled source relationships are omitted rather than inferred.
Main entry points: spotify_data_pipeline.py Spotify ETL DAG · crypto_exchange_pipeline.py Crypto ETL DAG · airlens_ingest_local.py Air-quality ingester · bikes.py Bike station producer
Components 29
AirLens platform
- Air-quality ingester
4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/ingestion/airlens_ingest_local.py - Air-quality APIs
- Local landing files
- Unity Catalog Volume
- Bronze processing
4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/notebooks/airlens_bronze.py - Silver processing
4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/notebooks/airlens_silver.py - DuckDB serving
4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/serving/build_duckdb.py - Streamlit dashboard
4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/app/streamlit_app.py - Airflow orchestration
4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/airflow/dags/airlens_dag.py
Orchestrated pipelines
- Spotify ETL DAG
5. Apache Airflow for Data Engineering/dags/spotify_data_pipeline.py - Spotify API
- Raw S3 data
- Album, artist, song transforms
5. Apache Airflow for Data Engineering/dags/spotify_data_pipeline.py - Transformed S3 data
- Crypto ETL DAG
5. Apache Airflow for Data Engineering/dags/crypto_exchange_pipeline.py - CoinGecko API
- Crypto transform
5. Apache Airflow for Data Engineering/dags/crypto_exchange_pipeline.py - Google Cloud Storage
- BigQuery
Kafka projects
- Bike station producer
6. Apache Kafka for Data Engineering/code/city-bikes/bikes/bikes.py - HTTP service
6. Apache Kafka for Data Engineering/code/city-bikes/services/http_service.py - GBFS station feeds
- Bike topic consumer
6. Apache Kafka for Data Engineering/code/city-bikes/consume.py - Kafka topics
Cloud and lakehouse
- ShopStream batch and streaming
4. Apache Spark with Databricks for Data Engineering/Project 3 - ShopStream Databricks Lakehouse (Batch + Streaming) - Stock-market data pipeline
6. Apache Kafka for Data Engineering/code/stockmarketdatapipeline - Ride Stream lakehouse
7. Cloud Computing for Data Engineer Projects/Ride Stream Real-Time Data Lakehouse - Custom Airflow extensions
5. Apache Airflow for Data Engineering/plugins - Spotify Spark pipeline
4. Apache Spark with Databricks for Data Engineering/Spotify Data Pipeline using Spark
Connections 15
- Air-quality ingester → Air-quality APIs: fetches readings
4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/ingestion/airlens_ingest_local.py - Air-quality ingester → Local landing files: writes JSON
4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/ingestion/airlens_ingest_local.py - Air-quality ingester → Unity Catalog Volume: uploads files
4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/ingestion/airlens_ingest_local.py - Spotify ETL DAG → Spotify API: fetches playlist
5. Apache Airflow for Data Engineering/dags/spotify_data_pipeline.py - Spotify ETL DAG → Raw S3 data: stores and reads raw
5. Apache Airflow for Data Engineering/dags/spotify_data_pipeline.py - Spotify ETL DAG → Album, artist, song transforms: dispatches transforms
5. Apache Airflow for Data Engineering/dags/spotify_data_pipeline.py - Spotify ETL DAG → Transformed S3 data: writes CSV outputs
5. Apache Airflow for Data Engineering/dags/spotify_data_pipeline.py - Crypto ETL DAG → CoinGecko API: fetches market data
5. Apache Airflow for Data Engineering/dags/crypto_exchange_pipeline.py - Crypto ETL DAG → Crypto transform: runs transform
5. Apache Airflow for Data Engineering/dags/crypto_exchange_pipeline.py - Crypto ETL DAG → Google Cloud Storage: uploads raw and transformed
5. Apache Airflow for Data Engineering/dags/crypto_exchange_pipeline.py - Crypto ETL DAG → BigQuery: loads transformed data
5. Apache Airflow for Data Engineering/dags/crypto_exchange_pipeline.py - Bike station producer → HTTP service: requests feeds
6. Apache Kafka for Data Engineering/code/city-bikes/bikes/bikes.py - HTTP service → GBFS station feeds: gets station data
- Bike station producer → Kafka topics: publishes station records
6. Apache Kafka for Data Engineering/code/city-bikes/bikes/bikes.py - Bike topic consumer → Kafka topics: consumes and commits
6. Apache Kafka for Data Engineering/code/city-bikes/consume.py