# anshlambagit/anshlambayoutube architecture

> Architecture diagram of the GitHub repository anshlambagit/anshlambayoutube, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anshlambagit/anshlambayoutube
- Repository: https://github.com/anshlambagit/anshlambayoutube

## Overview

This repository is a collection of data-engineering and Python learning materials, not one application. Its main demonstrated workflow is a declarative-pipeline tutorial: prepare example data, define transformations, then explore generated tables. The tree also shows separate Spark, Databricks, Fabric, dbt, and Python topics. README detail is minimal, and the sampled pipeline transformation files are entirely commented out, so their intended table flow is shown only as dashed. Other unsampled project internals are represented only at their exact enclosing directory paths.

Main entry points: [`Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations/sample_users_first_sdp_pipeline.py`](https://github.com/anshlambagit/anshlambayoutube/blob/main/Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations/sample_users_first_sdp_pipeline.py) (Sample Users), [`Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations/sample_aggregation_first_sdp_pipeline.py`](https://github.com/anshlambagit/anshlambayoutube/blob/main/Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations/sample_aggregation_first_sdp_pipeline.py) (Sample Aggregation), [`Spark_Declarative_Pipelines/SDP_Tutorial/source_prep.py`](https://github.com/anshlambagit/anshlambayoutube/blob/main/Spark_Declarative_Pipelines/SDP_Tutorial/source_prep.py) (Source Preparation), [`Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/explorations/sample_exploration.py`](https://github.com/anshlambagit/anshlambayoutube/blob/main/Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/explorations/sample_exploration.py) (Table Exploration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sdp["Declarative Pipelines"]
  node_sourceprep["Source Preparation<br/>[source_prep.py]"]
  node_sampleusers["Sample Users"]
  node_aggregation["Sample Aggregation"]
  node_pipelineutils["Pipeline Utilities<br/>[utils.py]"]
  node_exploration["Table Exploration"]
  node_sdpother["CDC And Quality"]
end

subgraph group_projects["Data Projects"]
  node_databricks["Databricks Projects"]
  node_dbt["DBT Masterclass"]
  node_fabric["Fabric Pipelines"]
  node_adf["ADF Projects"]
  node_sparkstream["Spark Streaming"]
end

subgraph group_python["Python Learning"]
  node_advancedpython["Advanced Python"]
  node_apiexample["Paginated API Example<br/>[advanced.py]"]
  node_asyncbasics["Async Example<br/>[1_basics.py]"]
  node_asyncprocess1["Process One<br/>[1_basics.py]"]
  node_asyncprocess2["Process Two<br/>[1_basics.py]"]
  node_pydantic["Pydantic Models"]
end

node_learner(("Learner"))
node_pokemon["PokéAPI"]

node_learner -->|"prepares examples"| node_sourceprep
node_learner -->|"studies transformations"| node_sampleusers
node_sourceprep -.->|"provides source tables"| node_sampleusers
node_sampleusers -.->|"reads sample table"| node_aggregation
node_aggregation -.->|"uses email helper"| node_pipelineutils
node_learner -->|"queries outputs"| node_exploration
node_exploration -.->|"reads materialized table"| node_aggregation
node_learner -->|"studies pipeline topics"| node_sdpother
node_learner -->|"studies project"| node_databricks
node_learner -->|"studies modeling"| node_dbt
node_learner -->|"studies pipeline"| node_fabric
node_learner -->|"studies orchestration"| node_adf
node_learner -->|"studies streaming"| node_sparkstream
node_learner -->|"studies Python topics"| node_advancedpython
node_apiexample -->|"requests pages"| node_pokemon
node_asyncbasics -->|"awaits process"| node_asyncprocess1
node_asyncprocess1 -->|"awaits process"| node_asyncprocess2
node_learner -->|"studies validation"| node_pydantic

click node_sourceprep "https://github.com/anshlambagit/anshlambayoutube/blob/main/Spark_Declarative_Pipelines/SDP_Tutorial/source_prep.py"
click node_sampleusers "https://github.com/anshlambagit/anshlambayoutube/blob/main/Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations/sample_users_first_sdp_pipeline.py"
click node_aggregation "https://github.com/anshlambagit/anshlambayoutube/blob/main/Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations/sample_aggregation_first_sdp_pipeline.py"
click node_pipelineutils "https://github.com/anshlambagit/anshlambayoutube/blob/main/Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/utilities/utils.py"
click node_exploration "https://github.com/anshlambagit/anshlambayoutube/blob/main/Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/explorations/sample_exploration.py"
click node_sdpother "https://github.com/anshlambagit/anshlambayoutube/tree/main/Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations"
click node_databricks "https://github.com/anshlambagit/anshlambayoutube/tree/main/Databricks%20End%20To%20End%20Project"
click node_dbt "https://github.com/anshlambagit/anshlambayoutube/tree/main/DBT_Masterclass"
click node_fabric "https://github.com/anshlambagit/anshlambayoutube/tree/main/FabricPipelines"
click node_adf "https://github.com/anshlambagit/anshlambayoutube/tree/main/ADF_PRO"
click node_sparkstream "https://github.com/anshlambagit/anshlambayoutube/tree/main/ApacheSparkStreaming"
click node_advancedpython "https://github.com/anshlambagit/anshlambayoutube/tree/main/Advanced_Python"
click node_apiexample "https://github.com/anshlambagit/anshlambayoutube/blob/main/Advanced_Python/CH-5APIs/advanced.py"
click node_asyncbasics "https://github.com/anshlambagit/anshlambayoutube/blob/main/Advanced_Python/Ch-3AsyncPython/1_basics.py"
click node_asyncprocess1 "https://github.com/anshlambagit/anshlambayoutube/blob/main/Advanced_Python/Ch-3AsyncPython/1_basics.py"
click node_asyncprocess2 "https://github.com/anshlambagit/anshlambayoutube/blob/main/Advanced_Python/Ch-3AsyncPython/1_basics.py"
click node_pydantic "https://github.com/anshlambagit/anshlambayoutube/tree/main/Advanced_Python/Ch-4Pydantic"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sourceprep,node_sampleusers,node_aggregation,node_pipelineutils,node_exploration,node_sdpother toneBlue
class node_databricks,node_dbt,node_fabric,node_adf,node_sparkstream toneAmber
class node_advancedpython,node_apiexample,node_asyncbasics,node_asyncprocess1,node_asyncprocess2,node_pydantic,node_pokemon toneMint
class node_learner toneIndigo
```

## Components

### Declarative Pipelines

- [Source Preparation](https://github.com/anshlambagit/anshlambayoutube/blob/main/Spark_Declarative_Pipelines/SDP_Tutorial/source_prep.py): `Spark_Declarative_Pipelines/SDP_Tutorial/source_prep.py`
- [Sample Users](https://github.com/anshlambagit/anshlambayoutube/blob/main/Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations/sample_users_first_sdp_pipeline.py): `Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations/sample_users_first_sdp_pipeline.py`
- [Sample Aggregation](https://github.com/anshlambagit/anshlambayoutube/blob/main/Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations/sample_aggregation_first_sdp_pipeline.py): `Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations/sample_aggregation_first_sdp_pipeline.py`
- [Pipeline Utilities](https://github.com/anshlambagit/anshlambayoutube/blob/main/Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/utilities/utils.py): `Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/utilities/utils.py`
- [Table Exploration](https://github.com/anshlambagit/anshlambayoutube/blob/main/Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/explorations/sample_exploration.py): `Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/explorations/sample_exploration.py`
- [CDC And Quality](https://github.com/anshlambagit/anshlambayoutube/tree/main/Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations): `Spark_Declarative_Pipelines/SDP_Tutorial/first_sdp_pipeline/transformations`

### Data Projects

- [Databricks Projects](https://github.com/anshlambagit/anshlambayoutube/tree/main/Databricks%20End%20To%20End%20Project): `Databricks End To End Project`
- [DBT Masterclass](https://github.com/anshlambagit/anshlambayoutube/tree/main/DBT_Masterclass): `DBT_Masterclass`
- [Fabric Pipelines](https://github.com/anshlambagit/anshlambayoutube/tree/main/FabricPipelines): `FabricPipelines`
- [ADF Projects](https://github.com/anshlambagit/anshlambayoutube/tree/main/ADF_PRO): `ADF_PRO`
- [Spark Streaming](https://github.com/anshlambagit/anshlambayoutube/tree/main/ApacheSparkStreaming): `ApacheSparkStreaming`

### Python Learning

- [Advanced Python](https://github.com/anshlambagit/anshlambayoutube/tree/main/Advanced_Python): `Advanced_Python`
- [Paginated API Example](https://github.com/anshlambagit/anshlambayoutube/blob/main/Advanced_Python/CH-5APIs/advanced.py): `Advanced_Python/CH-5APIs/advanced.py`
- [Async Example](https://github.com/anshlambagit/anshlambayoutube/blob/main/Advanced_Python/Ch-3AsyncPython/1_basics.py): `Advanced_Python/Ch-3AsyncPython/1_basics.py`
- [Process One](https://github.com/anshlambagit/anshlambayoutube/blob/main/Advanced_Python/Ch-3AsyncPython/1_basics.py): `Advanced_Python/Ch-3AsyncPython/1_basics.py`
- [Process Two](https://github.com/anshlambagit/anshlambayoutube/blob/main/Advanced_Python/Ch-3AsyncPython/1_basics.py): `Advanced_Python/Ch-3AsyncPython/1_basics.py`
- [Pydantic Models](https://github.com/anshlambagit/anshlambayoutube/tree/main/Advanced_Python/Ch-4Pydantic): `Advanced_Python/Ch-4Pydantic`

### Other components

- **Learner**
- **PokéAPI**

## Connections

- Learner → Source Preparation: prepares examples
- Learner → Sample Users: studies transformations
- Source Preparation → Sample Users: provides source tables
- Sample Users → Sample Aggregation: reads sample table
- Sample Aggregation → Pipeline Utilities: uses email helper
- Learner → Table Exploration: queries outputs
- Table Exploration → Sample Aggregation: reads materialized table
- Learner → CDC And Quality: studies pipeline topics
- Learner → Databricks Projects: studies project
- Learner → DBT Masterclass: studies modeling
- Learner → Fabric Pipelines: studies pipeline
- Learner → ADF Projects: studies orchestration
- Learner → Spark Streaming: studies streaming
- Learner → Advanced Python: studies Python topics
- Paginated API Example → PokéAPI: requests pages
- Async Example → Process One: awaits process
- Process One → Process Two: awaits process
- Learner → Pydantic Models: studies validation