# darshilparmar/datavidhya-data-engineering-course-resources architecture

> Architecture diagram of the GitHub repository darshilparmar/datavidhya-data-engineering-course-resources, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/darshilparmar/datavidhya-data-engineering-course-resources
- Repository: https://github.com/darshilparmar/datavidhya-data-engineering-course-resources

## Overview

This repository is a collection of independent data-engineering projects and examples, not one deployable application. Its workflows cover API extraction, orchestration, transformation, streaming, lakehouse storage, and serving. The graph highlights the sampled AirLens, Spotify, crypto, and City Bikes workflows; directory-level nodes represent other substantial project areas visible in the tree. The README is empty, and unsampled source relationships are omitted rather than inferred.

Main entry points: [`5. Apache Airflow for Data Engineering/dags/spotify_data_pipeline.py`](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/5.%20Apache%20Airflow%20for%20Data%20Engineering/dags/spotify_data_pipeline.py) (Spotify ETL DAG), [`5. Apache Airflow for Data Engineering/dags/crypto_exchange_pipeline.py`](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/5.%20Apache%20Airflow%20for%20Data%20Engineering/dags/crypto_exchange_pipeline.py) (Crypto ETL DAG), [`4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/ingestion/airlens_ingest_local.py`](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%204%20-%20AirLens%20Air%20Quality%20Platform%20(Modern%20Data%20Stack)/ingestion/airlens_ingest_local.py) (Air-quality ingester), [`6. Apache Kafka for Data Engineering/code/city-bikes/bikes/bikes.py`](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/6.%20Apache%20Kafka%20for%20Data%20Engineering/code/city-bikes/bikes/bikes.py) (Bike station producer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_airlens["AirLens platform"]
  node_ingester["Air-quality ingester"]
  node_air_sources(("Air-quality APIs"))
  node_landing[("Local landing files")]
  node_volume[("Unity Catalog Volume")]
  node_bronze["Bronze processing<br/>[airlens_bronze.py]"]
  node_silver["Silver processing<br/>[airlens_silver.py]"]
  node_duckdb["DuckDB serving<br/>[build_duckdb.py]"]
  node_air_app["Streamlit dashboard<br/>[streamlit_app.py]"]
  node_air_schedule["Airflow orchestration<br/>[airlens_dag.py]"]
end

subgraph group_orchestration["Orchestrated pipelines"]
  node_spotify_dag["Spotify ETL DAG"]
  node_spotify_api(("Spotify API"))
  node_spotify_raw[("Raw S3 data")]
  node_spotify_transform["Album, artist, song transforms"]
  node_spotify_outputs[("Transformed S3 data")]
  node_crypto_dag["Crypto ETL DAG"]
  node_coin_api(("CoinGecko API"))
  node_crypto_transform["Crypto transform"]
  node_gcs[("Google Cloud Storage")]
  node_bigquery[("BigQuery")]
end

subgraph group_kafka["Kafka projects"]
  node_bike_ingest["Bike station producer<br/>[bikes.py]"]
  node_bike_http["HTTP service<br/>[http_service.py]"]
  node_gbfs(("GBFS station feeds"))
  node_bike_consumer["Bike topic consumer<br/>[consume.py]"]
  node_kafka["Kafka topics"]
end

subgraph group_cloud["Cloud and lakehouse"]
  node_shopstream["ShopStream batch and streaming"]
  node_stock_pipeline["Stock-market data pipeline"]
  node_ride_pipeline["Ride Stream lakehouse"]
  node_custom_airflow["Custom Airflow extensions<br/>[plugins]"]
  node_spotify_spark["Spotify Spark pipeline"]
end

node_ingester -->|"fetches readings"| node_air_sources
node_ingester -->|"writes JSON"| node_landing
node_ingester -->|"uploads files"| node_volume
node_spotify_dag -->|"fetches playlist"| node_spotify_api
node_spotify_dag -->|"stores and reads raw"| node_spotify_raw
node_spotify_dag -->|"dispatches transforms"| node_spotify_transform
node_spotify_dag -->|"writes CSV outputs"| node_spotify_outputs
node_crypto_dag -->|"fetches market data"| node_coin_api
node_crypto_dag -->|"runs transform"| node_crypto_transform
node_crypto_dag -->|"uploads raw and transformed"| node_gcs
node_crypto_dag -->|"loads transformed data"| node_bigquery
node_bike_ingest -->|"requests feeds"| node_bike_http
node_bike_http -->|"gets station data"| node_gbfs
node_bike_ingest -->|"publishes station records"| node_kafka
node_bike_consumer -->|"consumes and commits"| node_kafka

click node_ingester "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%204%20-%20AirLens%20Air%20Quality%20Platform%20(Modern%20Data%20Stack)/ingestion/airlens_ingest_local.py"
click node_bronze "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%204%20-%20AirLens%20Air%20Quality%20Platform%20(Modern%20Data%20Stack)/notebooks/airlens_bronze.py"
click node_silver "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%204%20-%20AirLens%20Air%20Quality%20Platform%20(Modern%20Data%20Stack)/notebooks/airlens_silver.py"
click node_duckdb "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%204%20-%20AirLens%20Air%20Quality%20Platform%20(Modern%20Data%20Stack)/serving/build_duckdb.py"
click node_air_app "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%204%20-%20AirLens%20Air%20Quality%20Platform%20(Modern%20Data%20Stack)/app/streamlit_app.py"
click node_air_schedule "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%204%20-%20AirLens%20Air%20Quality%20Platform%20(Modern%20Data%20Stack)/airflow/dags/airlens_dag.py"
click node_spotify_dag "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/5.%20Apache%20Airflow%20for%20Data%20Engineering/dags/spotify_data_pipeline.py"
click node_spotify_transform "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/5.%20Apache%20Airflow%20for%20Data%20Engineering/dags/spotify_data_pipeline.py"
click node_crypto_dag "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/5.%20Apache%20Airflow%20for%20Data%20Engineering/dags/crypto_exchange_pipeline.py"
click node_crypto_transform "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/5.%20Apache%20Airflow%20for%20Data%20Engineering/dags/crypto_exchange_pipeline.py"
click node_bike_ingest "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/6.%20Apache%20Kafka%20for%20Data%20Engineering/code/city-bikes/bikes/bikes.py"
click node_bike_http "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/6.%20Apache%20Kafka%20for%20Data%20Engineering/code/city-bikes/services/http_service.py"
click node_bike_consumer "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/6.%20Apache%20Kafka%20for%20Data%20Engineering/code/city-bikes/consume.py"
click node_shopstream "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/tree/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%203%20-%20ShopStream%20Databricks%20Lakehouse%20(Batch%20%2B%20Streaming)"
click node_stock_pipeline "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/tree/main/6.%20Apache%20Kafka%20for%20Data%20Engineering/code/stockmarketdatapipeline"
click node_ride_pipeline "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/tree/main/7.%20Cloud%20Computing%20for%20Data%20Engineer%20Projects/Ride%20Stream%20Real-Time%20Data%20Lakehouse"
click node_custom_airflow "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/tree/main/5.%20Apache%20Airflow%20for%20Data%20Engineering/plugins"
click node_spotify_spark "https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/tree/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Spotify%20Data%20Pipeline%20using%20Spark"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingester,node_air_sources,node_landing,node_volume,node_bronze,node_silver,node_duckdb,node_air_app,node_air_schedule toneBlue
class node_spotify_dag,node_spotify_api,node_spotify_raw,node_spotify_transform,node_spotify_outputs,node_crypto_dag,node_coin_api,node_crypto_transform,node_gcs,node_bigquery toneAmber
class node_bike_ingest,node_bike_http,node_gbfs,node_bike_consumer,node_kafka toneMint
class node_shopstream,node_stock_pipeline,node_ride_pipeline,node_custom_airflow,node_spotify_spark toneRose
```

## Components

### AirLens platform

- [Air-quality ingester](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%204%20-%20AirLens%20Air%20Quality%20Platform%20(Modern%20Data%20Stack)/ingestion/airlens_ingest_local.py): `4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/ingestion/airlens_ingest_local.py`
- **Air-quality APIs**
- **Local landing files**
- **Unity Catalog Volume**
- [Bronze processing](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%204%20-%20AirLens%20Air%20Quality%20Platform%20(Modern%20Data%20Stack)/notebooks/airlens_bronze.py): `4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/notebooks/airlens_bronze.py`
- [Silver processing](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%204%20-%20AirLens%20Air%20Quality%20Platform%20(Modern%20Data%20Stack)/notebooks/airlens_silver.py): `4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/notebooks/airlens_silver.py`
- [DuckDB serving](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%204%20-%20AirLens%20Air%20Quality%20Platform%20(Modern%20Data%20Stack)/serving/build_duckdb.py): `4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/serving/build_duckdb.py`
- [Streamlit dashboard](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%204%20-%20AirLens%20Air%20Quality%20Platform%20(Modern%20Data%20Stack)/app/streamlit_app.py): `4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/app/streamlit_app.py`
- [Airflow orchestration](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%204%20-%20AirLens%20Air%20Quality%20Platform%20(Modern%20Data%20Stack)/airflow/dags/airlens_dag.py): `4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/airflow/dags/airlens_dag.py`

### Orchestrated pipelines

- [Spotify ETL DAG](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/5.%20Apache%20Airflow%20for%20Data%20Engineering/dags/spotify_data_pipeline.py): `5. Apache Airflow for Data Engineering/dags/spotify_data_pipeline.py`
- **Spotify API**
- **Raw S3 data**
- [Album, artist, song transforms](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/5.%20Apache%20Airflow%20for%20Data%20Engineering/dags/spotify_data_pipeline.py): `5. Apache Airflow for Data Engineering/dags/spotify_data_pipeline.py`
- **Transformed S3 data**
- [Crypto ETL DAG](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/5.%20Apache%20Airflow%20for%20Data%20Engineering/dags/crypto_exchange_pipeline.py): `5. Apache Airflow for Data Engineering/dags/crypto_exchange_pipeline.py`
- **CoinGecko API**
- [Crypto transform](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/5.%20Apache%20Airflow%20for%20Data%20Engineering/dags/crypto_exchange_pipeline.py): `5. Apache Airflow for Data Engineering/dags/crypto_exchange_pipeline.py`
- **Google Cloud Storage**
- **BigQuery**

### Kafka projects

- [Bike station producer](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/6.%20Apache%20Kafka%20for%20Data%20Engineering/code/city-bikes/bikes/bikes.py): `6. Apache Kafka for Data Engineering/code/city-bikes/bikes/bikes.py`
- [HTTP service](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/6.%20Apache%20Kafka%20for%20Data%20Engineering/code/city-bikes/services/http_service.py): `6. Apache Kafka for Data Engineering/code/city-bikes/services/http_service.py`
- **GBFS station feeds**
- [Bike topic consumer](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/blob/main/6.%20Apache%20Kafka%20for%20Data%20Engineering/code/city-bikes/consume.py): `6. Apache Kafka for Data Engineering/code/city-bikes/consume.py`
- **Kafka topics**

### Cloud and lakehouse

- [ShopStream batch and streaming](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/tree/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Project%203%20-%20ShopStream%20Databricks%20Lakehouse%20(Batch%20%2B%20Streaming)): `4. Apache Spark with Databricks for Data Engineering/Project 3 - ShopStream Databricks Lakehouse (Batch + Streaming)`
- [Stock-market data pipeline](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/tree/main/6.%20Apache%20Kafka%20for%20Data%20Engineering/code/stockmarketdatapipeline): `6. Apache Kafka for Data Engineering/code/stockmarketdatapipeline`
- [Ride Stream lakehouse](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/tree/main/7.%20Cloud%20Computing%20for%20Data%20Engineer%20Projects/Ride%20Stream%20Real-Time%20Data%20Lakehouse): `7. Cloud Computing for Data Engineer Projects/Ride Stream Real-Time Data Lakehouse`
- [Custom Airflow extensions](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/tree/main/5.%20Apache%20Airflow%20for%20Data%20Engineering/plugins): `5. Apache Airflow for Data Engineering/plugins`
- [Spotify Spark pipeline](https://github.com/darshilparmar/datavidhya-data-engineering-course-resources/tree/main/4.%20Apache%20Spark%20with%20Databricks%20for%20Data%20Engineering/Spotify%20Data%20Pipeline%20using%20Spark): `4. Apache Spark with Databricks for Data Engineering/Spotify Data Pipeline using Spark`

## Connections

- Air-quality ingester → Air-quality APIs: fetches readings (evidence: `4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/ingestion/airlens_ingest_local.py`)
- Air-quality ingester → Local landing files: writes JSON (evidence: `4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/ingestion/airlens_ingest_local.py`)
- Air-quality ingester → Unity Catalog Volume: uploads files (evidence: `4. Apache Spark with Databricks for Data Engineering/Project 4 - AirLens Air Quality Platform (Modern Data Stack)/ingestion/airlens_ingest_local.py`)
- Spotify ETL DAG → Spotify API: fetches playlist (evidence: `5. Apache Airflow for Data Engineering/dags/spotify_data_pipeline.py`)
- Spotify ETL DAG → Raw S3 data: stores and reads raw (evidence: `5. Apache Airflow for Data Engineering/dags/spotify_data_pipeline.py`)
- Spotify ETL DAG → Album, artist, song transforms: dispatches transforms (evidence: `5. Apache Airflow for Data Engineering/dags/spotify_data_pipeline.py`)
- Spotify ETL DAG → Transformed S3 data: writes CSV outputs (evidence: `5. Apache Airflow for Data Engineering/dags/spotify_data_pipeline.py`)
- Crypto ETL DAG → CoinGecko API: fetches market data (evidence: `5. Apache Airflow for Data Engineering/dags/crypto_exchange_pipeline.py`)
- Crypto ETL DAG → Crypto transform: runs transform (evidence: `5. Apache Airflow for Data Engineering/dags/crypto_exchange_pipeline.py`)
- Crypto ETL DAG → Google Cloud Storage: uploads raw and transformed (evidence: `5. Apache Airflow for Data Engineering/dags/crypto_exchange_pipeline.py`)
- Crypto ETL DAG → BigQuery: loads transformed data (evidence: `5. Apache Airflow for Data Engineering/dags/crypto_exchange_pipeline.py`)
- Bike station producer → HTTP service: requests feeds (evidence: `6. Apache Kafka for Data Engineering/code/city-bikes/bikes/bikes.py`)
- HTTP service → GBFS station feeds: gets station data
- Bike station producer → Kafka topics: publishes station records (evidence: `6. Apache Kafka for Data Engineering/code/city-bikes/bikes/bikes.py`)
- Bike topic consumer → Kafka topics: consumes and commits (evidence: `6. Apache Kafka for Data Engineering/code/city-bikes/consume.py`)