How anshlambagit/anshlambayoutube works
This repository is a collection of data-engineering and Python learning materials, not one application. Its main demonstrated workflow is a declarative-pipeline tutorial: prepare example data, define transformations, then explore generated tables. The tree also shows separate Spark, Databricks, Fabric, dbt, and Python topics. README detail is minimal, and the sampled pipeline transformation files are entirely commented out, so their intended table flow is shown only as dashed. Other unsampled project internals are represented only at their exact enclosing directory paths.
Main entry points: sample_users_first_sdp_pipeline.py Sample Users · sample_aggregation_first_sdp_pipeline.py Sample Aggregation · source_prep.py Source Preparation · sample_exploration.py Table Exploration
Components 19
Declarative Pipelines
- Source Preparation
Spark_Declarative_Pipelines/SDP_Tutorial/source_prep.py - Sample Users
Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations/sample_users_first_sdp_pipeline.py - Sample Aggregation
Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations/sample_aggregation_first_sdp_pipeline.py - Pipeline Utilities
Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/utilities/utils.py - Table Exploration
Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/explorations/sample_exploration.py - CDC And Quality
Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations
Data Projects
- Databricks Projects
Databricks End To End Project - DBT Masterclass
DBT_Masterclass - Fabric Pipelines
FabricPipelines - ADF Projects
ADF_PRO - Spark Streaming
ApacheSparkStreaming
Python Learning
- Advanced Python
Advanced_Python - Paginated API Example
Advanced_Python/CH-5APIs/advanced.py - Async Example
Advanced_Python/Ch-3AsyncPython/1_basics.py - Process One
Advanced_Python/Ch-3AsyncPython/1_basics.py - Process Two
Advanced_Python/Ch-3AsyncPython/1_basics.py - Pydantic Models
Advanced_Python/Ch-4Pydantic
Other components
- Learner
- PokéAPI
Connections 18
- Learner → Source Preparation: prepares examples
- Learner → Sample Users: studies transformations
- Source Preparation → Sample Users: provides source tables
- Sample Users → Sample Aggregation: reads sample table
- Sample Aggregation → Pipeline Utilities: uses email helper
- Learner → Table Exploration: queries outputs
- Table Exploration → Sample Aggregation: reads materialized table
- Learner → CDC And Quality: studies pipeline topics
- Learner → Databricks Projects: studies project
- Learner → DBT Masterclass: studies modeling
- Learner → Fabric Pipelines: studies pipeline
- Learner → ADF Projects: studies orchestration
- Learner → Spark Streaming: studies streaming
- Learner → Advanced Python: studies Python topics
- Paginated API Example → PokéAPI: requests pages
- Async Example → Process One: awaits process
- Process One → Process Two: awaits process
- Learner → Pydantic Models: studies validation