How prabhakarandatascience/data_practice works
This repository is a collection of data-engineering learning notebooks, not a single deployable application. Its main learning workflow covers ingestion, Delta Lake storage, medallion transformations, declarative pipelines, and job orchestration. The clearest executable example is the pipeline source file, which reads volume files into bronze, transforms bronze into silver, and aggregates silver into gold. Other notebooks document ingestion methods, governance, and orchestration. The graph reflects shown code and explicit lesson text; unsampled notebooks are represented only where their paths and topic are clear.
Main entry points: Pipeline - 1.py Bronze table · 02 - Ingesting Data into Delta Lake.py Delta tables and volumes · 02 - Creating and Managing Spark Declarative Pipelines.py Pipeline management · 01 - Data Engineering Basics.py Engineering basics
Components 26
Learning foundations
- Engineering basics
Data Engineering/M-01 -- Introduction/01 - Data Engineering Basics.py - Lakeflow overview
Data Engineering/M-01 -- Introduction/02 - Intro to Lakeflow.py
Ingestion and storage
- Lakeflow Connect
Data Engineering/M-02 -- Ingestion and Transformation/01 - Ingestion with Lakeflow Connect.py - Delta ingestion
Data Engineering/M-02 -- Ingestion and Transformation/02 - Ingesting Data into Delta Lake.py - Delta tables and volumes
Data Engineering/M-02 -- Ingestion and Transformation/02 - Ingesting Data into Delta Lake.py
Transformations
- Medallion concepts
Data Engineering/M-02 -- Ingestion and Transformation/03 - Data Transformation with the Medallion Architecture.py - Medallion transformations
Data Engineering/M-02 -- Ingestion and Transformation/04 - Transforming Data Using the Medallion Architecture.py - Bronze table
Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py - Silver table
Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py - Gold aggregate
Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py
Pipelines and operations
- Declarative ETL
Data Engineering/M-03 -- Pipelines/01 - ETL with Lakeflow Spark Declarative Pipelines.py - Pipeline management
Data Engineering/M-03 -- Pipelines/02 - Creating and Managing Spark Declarative Pipelines.py - Pipeline definitions
Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py - Lakeflow Jobs
Data Engineering/M-04 -- Orchestration/01 - Orchestration with Lakeflow Jobs.py
Practice notebooks
- Ingestion lab
Data Engineering/M-02 -- Ingestion and Transformation/05 - Ingesting and Manipulation Lab.py - PySpark practice
Pyspark basics.ipynb - DataFrame practice
spark1/DataFrame.ipynb - PySpark exercises
spark1/Pyspark Practice.ipynb - SQL exercises
SQL/01 sql_practice.ipynb - Intermediate SQL
SQL/02 Intermediate sql day 2.ipynb - CTE and windows
SQL/03 CTE and Window function.ipynb - SQL challenge
SQL/04 Adv_Data_Enginnering_sql_challenge.ipynb - Window and dates
SQL/05 Positional Window Functions & Date Manipulation!.ipynb - Database objects
database object.ipynb - SQL transformation
SDP_Practice_with_sql/SDP_Practice_with_sql/transformations/my_transformation.sql
Other components
- Data engineer
Connections 14
- Data engineer → Engineering basics: studies
Data Engineering/M-01 -- Introduction/01 - Data Engineering Basics.py - Engineering basics → Lakeflow overview: introduces
Data Engineering/M-01 -- Introduction/01 - Data Engineering Basics.py - Lakeflow overview → Lakeflow Connect: describes
Data Engineering/M-01 -- Introduction/02 - Intro to Lakeflow.py - Lakeflow Connect → Delta ingestion: ingests via
Data Engineering/M-02 -- Ingestion and Transformation/01 - Ingestion with Lakeflow Connect.py - Delta ingestion → Delta tables and volumes: creates tables
Data Engineering/M-02 -- Ingestion and Transformation/02 - Ingesting Data into Delta Lake.py - Medallion concepts → Medallion transformations: demonstrates
Data Engineering/M-02 -- Ingestion and Transformation/03 - Data Transformation with the Medallion Architecture.py - Medallion transformations → Delta tables and volumes: reads and writes
Data Engineering/M-02 -- Ingestion and Transformation/04 - Transforming Data Using the Medallion Architecture.py - Pipeline management → Pipeline definitions: uses starter files
Data Engineering/M-03 -- Pipelines/02 - Creating and Managing Spark Declarative Pipelines.py - Pipeline definitions → Bronze table: loads files
Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py - Bronze table → Silver table: streams and cleans
Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py - Silver table → Gold aggregate: aggregates
Data Engineering/M-03 -- Pipelines/Pipeline Files/Pipeline - 1.py - Declarative ETL → Bronze table: describes layers
Data Engineering/M-03 -- Pipelines/01 - ETL with Lakeflow Spark Declarative Pipelines.py - Pipeline management → Lakeflow Jobs: schedules pipeline
Data Engineering/M-03 -- Pipelines/02 - Creating and Managing Spark Declarative Pipelines.py - Ingestion lab → Delta tables and volumes: loads and modifies
Data Engineering/M-02 -- Ingestion and Transformation/05 - Ingesting and Manipulation Lab.py