# prabhakarandatascience/data_practice architecture

> Architecture diagram of the GitHub repository prabhakarandatascience/data_practice, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/prabhakarandatascience/data_practice
- Repository: https://github.com/prabhakarandatascience/data_practice

## Overview

This repository is a collection of data-engineering learning notebooks, not a single deployable application. Its main learning workflow covers ingestion, Delta Lake storage, medallion transformations, declarative pipelines, and job orchestration. The clearest executable example is the pipeline source file, which reads volume files into bronze, transforms bronze into silver, and aggregates silver into gold. Other notebooks document ingestion methods, governance, and orchestration. The graph reflects shown code and explicit lesson text; unsampled notebooks are represented only where their paths and topic are clear.

Main entry points: [`Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py`](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-03%20--%20Pipelines/Pipeline%20Files/Pipeline%20-%201.py) (Bronze table), [`Data Engineering/M-02 -- Ingestion and Transformation/02 - Ingesting Data into Delta Lake.py`](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-02%20--%20Ingestion%20and%20Transformation/02%20-%20Ingesting%20Data%20into%20Delta%20Lake.py) (Delta tables and volumes), [`Data Engineering/M-03 -- Pipelines/02 - Creating and Managing Spark Declarative Pipelines.py`](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-03%20--%20Pipelines/02%20-%20Creating%20and%20Managing%20Spark%20Declarative%20Pipelines.py) (Pipeline management), [`Data Engineering/M-01 -- Introduction/01 - Data Engineering Basics.py`](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-01%20--%20Introduction/01%20-%20Data%20Engineering%20Basics.py) (Engineering basics)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_learning["Learning foundations"]
  node_basics["Engineering basics"]
  node_lakeflow["Lakeflow overview"]
end

subgraph group_ingest["Ingestion and storage"]
  node_connect["Lakeflow Connect"]
  node_delta_ingest["Delta ingestion"]
  node_delta_store[("Delta tables and volumes")]
end

subgraph group_transform["Transformations"]
  node_medallion_concepts["Medallion concepts"]
  node_medallion_lab["Medallion transformations"]
  node_bronze[("Bronze table<br/>[Pipeline - 1.py]")]
  node_silver[("Silver table<br/>[Pipeline - 1.py]")]
  node_gold[("Gold aggregate<br/>[Pipeline - 1.py]")]
end

subgraph group_operate["Pipelines and operations"]
  node_sdp_intro["Declarative ETL"]
  node_pipeline_lesson["Pipeline management"]
  node_pipeline_code["Pipeline definitions<br/>[Pipeline - 1.py]"]
  node_jobs["Lakeflow Jobs"]
end

subgraph group_practice["Practice notebooks"]
  node_ingest_lab["Ingestion lab"]
  node_pyspark["PySpark practice"]
  node_dataframe["DataFrame practice<br/>[DataFrame.ipynb]"]
  node_spark_practice["PySpark exercises"]
  node_sql_practice["SQL exercises"]
  node_sql_intermediate["Intermediate SQL"]
  node_sql_windows["CTE and windows"]
  node_sql_challenge["SQL challenge"]
  node_sql_dates["Window and dates"]
  node_db_objects["Database objects"]
  node_sql_transform["SQL transformation"]
end

node_learner(("Data engineer"))

node_learner -->|"studies"| node_basics
node_basics -->|"introduces"| node_lakeflow
node_lakeflow -->|"describes"| node_connect
node_connect -->|"ingests via"| node_delta_ingest
node_delta_ingest -->|"creates tables"| node_delta_store
node_medallion_concepts -->|"demonstrates"| node_medallion_lab
node_medallion_lab -->|"reads and writes"| node_delta_store
node_pipeline_lesson -->|"uses starter files"| node_pipeline_code
node_pipeline_code -->|"loads files"| node_bronze
node_bronze -->|"streams and cleans"| node_silver
node_silver -->|"aggregates"| node_gold
node_sdp_intro -->|"describes layers"| node_bronze
node_pipeline_lesson -->|"schedules pipeline"| node_jobs
node_ingest_lab -->|"loads and modifies"| node_delta_store

click node_basics "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-01%20--%20Introduction/01%20-%20Data%20Engineering%20Basics.py"
click node_lakeflow "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-01%20--%20Introduction/02%20-%20Intro%20to%20Lakeflow.py"
click node_connect "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-02%20--%20Ingestion%20and%20Transformation/01%20-%20Ingestion%20with%20Lakeflow%20Connect.py"
click node_delta_ingest "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-02%20--%20Ingestion%20and%20Transformation/02%20-%20Ingesting%20Data%20into%20Delta%20Lake.py"
click node_delta_store "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-02%20--%20Ingestion%20and%20Transformation/02%20-%20Ingesting%20Data%20into%20Delta%20Lake.py"
click node_medallion_concepts "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-02%20--%20Ingestion%20and%20Transformation/03%20-%20Data%20Transformation%20with%20the%20Medallion%20Architecture.py"
click node_medallion_lab "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-02%20--%20Ingestion%20and%20Transformation/04%20-%20Transforming%20Data%20Using%20the%20Medallion%20Architecture.py"
click node_bronze "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-03%20--%20Pipelines/Pipeline%20Files/Pipeline%20-%201.py"
click node_silver "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-03%20--%20Pipelines/Pipeline%20Files/Pipeline%20-%201.py"
click node_gold "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-03%20--%20Pipelines/Pipeline%20Files/Pipeline%20-%201.py"
click node_ingest_lab "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-02%20--%20Ingestion%20and%20Transformation/05%20-%20Ingesting%20and%20Manipulation%20Lab.py"
click node_sdp_intro "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-03%20--%20Pipelines/01%20-%20ETL%20with%20Lakeflow%20Spark%20Declarative%20Pipelines.py"
click node_pipeline_lesson "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-03%20--%20Pipelines/02%20-%20Creating%20and%20Managing%20Spark%20Declarative%20Pipelines.py"
click node_pipeline_code "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-03%20--%20Pipelines/Pipeline%20Files/Pipeline%20-%201.py"
click node_jobs "https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-04%20--%20Orchestration/01%20-%20Orchestration%20with%20Lakeflow%20Jobs.py"
click node_pyspark "https://github.com/prabhakarandatascience/data_practice/blob/main/Pyspark%20basics.ipynb"
click node_dataframe "https://github.com/prabhakarandatascience/data_practice/blob/main/spark1/DataFrame.ipynb"
click node_spark_practice "https://github.com/prabhakarandatascience/data_practice/blob/main/spark1/Pyspark%20Practice.ipynb"
click node_sql_practice "https://github.com/prabhakarandatascience/data_practice/blob/main/SQL/01%20sql_practice.ipynb"
click node_sql_intermediate "https://github.com/prabhakarandatascience/data_practice/blob/main/SQL/02%20Intermediate%20sql%20day%202.ipynb"
click node_sql_windows "https://github.com/prabhakarandatascience/data_practice/blob/main/SQL/03%20CTE%20and%20Window%20function.ipynb"
click node_sql_challenge "https://github.com/prabhakarandatascience/data_practice/blob/main/SQL/04%20Adv_Data_Enginnering_sql_challenge.ipynb"
click node_sql_dates "https://github.com/prabhakarandatascience/data_practice/blob/main/SQL/05%20Positional%20Window%20Functions%20%26%20Date%20Manipulation!.ipynb"
click node_db_objects "https://github.com/prabhakarandatascience/data_practice/blob/main/database%20object.ipynb"
click node_sql_transform "https://github.com/prabhakarandatascience/data_practice/blob/main/SDP_Practice_with_sql/SDP_Practice_with_sql/transformations/my_transformation.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_basics,node_lakeflow toneBlue
class node_connect,node_delta_ingest,node_delta_store toneAmber
class node_medallion_concepts,node_medallion_lab,node_bronze,node_silver,node_gold toneMint
class node_sdp_intro,node_pipeline_lesson,node_pipeline_code,node_jobs toneRose
class node_ingest_lab,node_pyspark,node_dataframe,node_spark_practice,node_sql_practice,node_sql_intermediate,node_sql_windows,node_sql_challenge,node_sql_dates,node_db_objects,node_sql_transform,node_learner toneIndigo
```

## Components

### Learning foundations

- [Engineering basics](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-01%20--%20Introduction/01%20-%20Data%20Engineering%20Basics.py): `Data Engineering/M-01 -- Introduction/01 - Data Engineering Basics.py`
- [Lakeflow overview](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-01%20--%20Introduction/02%20-%20Intro%20to%20Lakeflow.py): `Data Engineering/M-01 -- Introduction/02 - Intro to Lakeflow.py`

### Ingestion and storage

- [Lakeflow Connect](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-02%20--%20Ingestion%20and%20Transformation/01%20-%20Ingestion%20with%20Lakeflow%20Connect.py): `Data Engineering/M-02 -- Ingestion and Transformation/01 - Ingestion with Lakeflow Connect.py`
- [Delta ingestion](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-02%20--%20Ingestion%20and%20Transformation/02%20-%20Ingesting%20Data%20into%20Delta%20Lake.py): `Data Engineering/M-02 -- Ingestion and Transformation/02 - Ingesting Data into Delta Lake.py`
- [Delta tables and volumes](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-02%20--%20Ingestion%20and%20Transformation/02%20-%20Ingesting%20Data%20into%20Delta%20Lake.py): `Data Engineering/M-02 -- Ingestion and Transformation/02 - Ingesting Data into Delta Lake.py`

### Transformations

- [Medallion concepts](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-02%20--%20Ingestion%20and%20Transformation/03%20-%20Data%20Transformation%20with%20the%20Medallion%20Architecture.py): `Data Engineering/M-02 -- Ingestion and Transformation/03 - Data Transformation with the Medallion Architecture.py`
- [Medallion transformations](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-02%20--%20Ingestion%20and%20Transformation/04%20-%20Transforming%20Data%20Using%20the%20Medallion%20Architecture.py): `Data Engineering/M-02 -- Ingestion and Transformation/04 - Transforming Data Using the Medallion Architecture.py`
- [Bronze table](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-03%20--%20Pipelines/Pipeline%20Files/Pipeline%20-%201.py): `Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py`
- [Silver table](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-03%20--%20Pipelines/Pipeline%20Files/Pipeline%20-%201.py): `Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py`
- [Gold aggregate](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-03%20--%20Pipelines/Pipeline%20Files/Pipeline%20-%201.py): `Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py`

### Pipelines and operations

- [Declarative ETL](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-03%20--%20Pipelines/01%20-%20ETL%20with%20Lakeflow%20Spark%20Declarative%20Pipelines.py): `Data Engineering/M-03 -- Pipelines/01 - ETL with Lakeflow Spark Declarative Pipelines.py`
- [Pipeline management](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-03%20--%20Pipelines/02%20-%20Creating%20and%20Managing%20Spark%20Declarative%20Pipelines.py): `Data Engineering/M-03 -- Pipelines/02 - Creating and Managing Spark Declarative Pipelines.py`
- [Pipeline definitions](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-03%20--%20Pipelines/Pipeline%20Files/Pipeline%20-%201.py): `Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py`
- [Lakeflow Jobs](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-04%20--%20Orchestration/01%20-%20Orchestration%20with%20Lakeflow%20Jobs.py): `Data Engineering/M-04 -- Orchestration/01 - Orchestration with Lakeflow Jobs.py`

### Practice notebooks

- [Ingestion lab](https://github.com/prabhakarandatascience/data_practice/blob/main/Data%20Engineering/M-02%20--%20Ingestion%20and%20Transformation/05%20-%20Ingesting%20and%20Manipulation%20Lab.py): `Data Engineering/M-02 -- Ingestion and Transformation/05 - Ingesting and Manipulation Lab.py`
- [PySpark practice](https://github.com/prabhakarandatascience/data_practice/blob/main/Pyspark%20basics.ipynb): `Pyspark basics.ipynb`
- [DataFrame practice](https://github.com/prabhakarandatascience/data_practice/blob/main/spark1/DataFrame.ipynb): `spark1/DataFrame.ipynb`
- [PySpark exercises](https://github.com/prabhakarandatascience/data_practice/blob/main/spark1/Pyspark%20Practice.ipynb): `spark1/Pyspark Practice.ipynb`
- [SQL exercises](https://github.com/prabhakarandatascience/data_practice/blob/main/SQL/01%20sql_practice.ipynb): `SQL/01 sql_practice.ipynb`
- [Intermediate SQL](https://github.com/prabhakarandatascience/data_practice/blob/main/SQL/02%20Intermediate%20sql%20day%202.ipynb): `SQL/02 Intermediate sql day 2.ipynb`
- [CTE and windows](https://github.com/prabhakarandatascience/data_practice/blob/main/SQL/03%20CTE%20and%20Window%20function.ipynb): `SQL/03 CTE and Window function.ipynb`
- [SQL challenge](https://github.com/prabhakarandatascience/data_practice/blob/main/SQL/04%20Adv_Data_Enginnering_sql_challenge.ipynb): `SQL/04 Adv_Data_Enginnering_sql_challenge.ipynb`
- [Window and dates](https://github.com/prabhakarandatascience/data_practice/blob/main/SQL/05%20Positional%20Window%20Functions%20%26%20Date%20Manipulation!.ipynb): `SQL/05 Positional Window Functions & Date Manipulation!.ipynb`
- [Database objects](https://github.com/prabhakarandatascience/data_practice/blob/main/database%20object.ipynb): `database object.ipynb`
- [SQL transformation](https://github.com/prabhakarandatascience/data_practice/blob/main/SDP_Practice_with_sql/SDP_Practice_with_sql/transformations/my_transformation.sql): `SDP_Practice_with_sql/SDP_Practice_with_sql/transformations/my_transformation.sql`

### Other components

- **Data engineer**

## Connections

- Data engineer → Engineering basics: studies (evidence: `Data Engineering/M-01 -- Introduction/01 - Data Engineering Basics.py`)
- Engineering basics → Lakeflow overview: introduces (evidence: `Data Engineering/M-01 -- Introduction/01 - Data Engineering Basics.py`)
- Lakeflow overview → Lakeflow Connect: describes (evidence: `Data Engineering/M-01 -- Introduction/02 - Intro to Lakeflow.py`)
- Lakeflow Connect → Delta ingestion: ingests via (evidence: `Data Engineering/M-02 -- Ingestion and Transformation/01 - Ingestion with Lakeflow Connect.py`)
- Delta ingestion → Delta tables and volumes: creates tables (evidence: `Data Engineering/M-02 -- Ingestion and Transformation/02 - Ingesting Data into Delta Lake.py`)
- Medallion concepts → Medallion transformations: demonstrates (evidence: `Data Engineering/M-02 -- Ingestion and Transformation/03 - Data Transformation with the Medallion Architecture.py`)
- Medallion transformations → Delta tables and volumes: reads and writes (evidence: `Data Engineering/M-02 -- Ingestion and Transformation/04 - Transforming Data Using the Medallion Architecture.py`)
- Pipeline management → Pipeline definitions: uses starter files (evidence: `Data Engineering/M-03 -- Pipelines/02 - Creating and Managing Spark Declarative Pipelines.py`)
- Pipeline definitions → Bronze table: loads files (evidence: `Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py`)
- Bronze table → Silver table: streams and cleans (evidence: `Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py`)
- Silver table → Gold aggregate: aggregates (evidence: `Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py`)
- Declarative ETL → Bronze table: describes layers (evidence: `Data Engineering/M-03 -- Pipelines/01 - ETL with Lakeflow Spark Declarative Pipelines.py`)
- Pipeline management → Lakeflow Jobs: schedules pipeline (evidence: `Data Engineering/M-03 -- Pipelines/02 - Creating and Managing Spark Declarative Pipelines.py`)
- Ingestion lab → Delta tables and volumes: loads and modifies (evidence: `Data Engineering/M-02 -- Ingestion and Transformation/05 - Ingesting and Manipulation Lab.py`)
