# jan-dyndor/life_etl architecture

> Architecture diagram of the GitHub repository jan-dyndor/life_etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jan-dyndor/life_etl
- Repository: https://github.com/jan-dyndor/life_etl

## Overview

Life_ETL is a Databricks pipeline for daily NBP foreign-exchange data, with a Streamlit dashboard for inspecting the exported Gold dataset. The documented workflow runs from scheduled extraction through Bronze, Silver and Gold processing to GitHub export and notification; dashboard users load, filter and visualize the published data. Notebook names and the README support the ETL stages, but their internal code was not sampled, so their detailed transformations and wiring are not inferred. Dashboard behavior is supported by the sampled app and analytics sources.

Main entry points: [`ETL/download_data_from_npb_API.ipynb`](https://github.com/jan-dyndor/life_etl/blob/main/ETL/download_data_from_npb_API.ipynb) (NBP extraction), [`pages/analytics_page.py`](https://github.com/jan-dyndor/life_etl/blob/main/pages/analytics_page.py) (Analytics page), [`streamlit_app/functions/functions.py`](https://github.com/jan-dyndor/life_etl/blob/main/streamlit_app/functions/functions.py) (Load and filter), [`ETL/Bronze_Layer_Autoloader.ipynb`](https://github.com/jan-dyndor/life_etl/blob/main/ETL/Bronze_Layer_Autoloader.ipynb) (Bronze layer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_etl["Databricks ETL"]
  node_pipeline_config["Job orchestration"]
  node_extract["NBP extraction"]
  node_eda["Bronze exploration<br/>[EDA_bronze.ipynb]"]
  node_export["GitHub export"]
end

subgraph group_layers["Medallion Data"]
  node_source[("Source data")]
  node_bronze["Bronze ingestion"]
  node_bronze_data[("Bronze layer")]
  node_silver["Silver processing"]
  node_silver_data[("Silver layer")]
  node_gold["Gold processing"]
  node_gold_data[("Gold layer")]
end

subgraph group_dashboard["Streamlit Analytics"]
  node_app["Project overview<br/>[app.py]"]
  node_analytics["Analytics page<br/>[analytics_page.py]"]
  node_data_functions["Load and filter<br/>[functions.py]"]
  node_filtered_data["Filtered FX data<br/>[functions.py]"]
  node_charts["Charts and analysis<br/>[analytics_page.py]"]
end

node_scheduler(("Daily scheduler"))
node_databricks["Databricks job"]
node_nbp_api["NBP API"]
node_github[("GitHub dataset")]
node_email["Email notification"]
node_user(("Dashboard user"))

node_scheduler -->|"triggers daily"| node_databricks
node_databricks -->|"runs job"| node_pipeline_config
node_pipeline_config -->|"orchestrates"| node_extract
node_extract -->|"fetches rates"| node_nbp_api
node_extract -->|"writes raw data"| node_source
node_bronze -->|"reads raw data"| node_source
node_bronze -->|"writes layer"| node_bronze_data
node_eda -.->|"explores"| node_bronze_data
node_silver -->|"reads layer"| node_bronze_data
node_silver -->|"writes layer"| node_silver_data
node_gold -->|"reads layer"| node_silver_data
node_gold -->|"writes layer"| node_gold_data
node_export -->|"reads dataset"| node_gold_data
node_export -->|"exports dataset"| node_github
node_databricks -.->|"notifies on success"| node_email
node_user -->|"opens"| node_app
node_user -->|"selects page"| node_analytics
node_analytics -->|"loads data"| node_data_functions
node_data_functions -->|"requests CSV"| node_github
node_data_functions -->|"returns filtered data"| node_filtered_data
node_analytics -->|"renders analysis"| node_charts
node_filtered_data -->|"feeds charts"| node_charts

click node_pipeline_config "https://github.com/jan-dyndor/life_etl/blob/main/ETL/pipeline_config.ipynb"
click node_extract "https://github.com/jan-dyndor/life_etl/blob/main/ETL/download_data_from_npb_API.ipynb"
click node_source "https://github.com/jan-dyndor/life_etl/blob/main/ETL/download_data_from_npb_API.ipynb"
click node_bronze "https://github.com/jan-dyndor/life_etl/blob/main/ETL/Bronze_Layer_Autoloader.ipynb"
click node_bronze_data "https://github.com/jan-dyndor/life_etl/blob/main/ETL/Bronze_Layer_Autoloader.ipynb"
click node_eda "https://github.com/jan-dyndor/life_etl/blob/main/ETL/EDA_bronze.ipynb"
click node_silver "https://github.com/jan-dyndor/life_etl/blob/main/ETL/Silver_Layer_Autoloader.ipynb"
click node_silver_data "https://github.com/jan-dyndor/life_etl/blob/main/ETL/Silver_Layer_Autoloader.ipynb"
click node_gold "https://github.com/jan-dyndor/life_etl/blob/main/ETL/Gold_Autoloader.ipynb"
click node_gold_data "https://github.com/jan-dyndor/life_etl/blob/main/ETL/Gold_Autoloader.ipynb"
click node_export "https://github.com/jan-dyndor/life_etl/blob/main/ETL/Export_Data_Into_Volue_And_Send_To_GitHub.ipynb"
click node_app "https://github.com/jan-dyndor/life_etl/blob/main/app.py"
click node_analytics "https://github.com/jan-dyndor/life_etl/blob/main/pages/analytics_page.py"
click node_data_functions "https://github.com/jan-dyndor/life_etl/blob/main/streamlit_app/functions/functions.py"
click node_filtered_data "https://github.com/jan-dyndor/life_etl/blob/main/streamlit_app/functions/functions.py"
click node_charts "https://github.com/jan-dyndor/life_etl/blob/main/pages/analytics_page.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline_config,node_extract,node_eda,node_export,node_user toneBlue
class node_source,node_bronze,node_bronze_data,node_silver,node_silver_data,node_gold,node_gold_data,node_github toneAmber
class node_app,node_analytics,node_data_functions,node_filtered_data,node_charts,node_nbp_api toneMint
class node_scheduler toneRose
class node_databricks,node_email toneIndigo
```

## Components

### Databricks ETL

- [Job orchestration](https://github.com/jan-dyndor/life_etl/blob/main/ETL/pipeline_config.ipynb): `ETL/pipeline_config.ipynb`
- [NBP extraction](https://github.com/jan-dyndor/life_etl/blob/main/ETL/download_data_from_npb_API.ipynb): `ETL/download_data_from_npb_API.ipynb`
- [Bronze exploration](https://github.com/jan-dyndor/life_etl/blob/main/ETL/EDA_bronze.ipynb): `ETL/EDA_bronze.ipynb`
- [GitHub export](https://github.com/jan-dyndor/life_etl/blob/main/ETL/Export_Data_Into_Volue_And_Send_To_GitHub.ipynb): `ETL/Export_Data_Into_Volue_And_Send_To_GitHub.ipynb`

### Medallion Data

- [Source data](https://github.com/jan-dyndor/life_etl/blob/main/ETL/download_data_from_npb_API.ipynb): `ETL/download_data_from_npb_API.ipynb`
- [Bronze ingestion](https://github.com/jan-dyndor/life_etl/blob/main/ETL/Bronze_Layer_Autoloader.ipynb): `ETL/Bronze_Layer_Autoloader.ipynb`
- [Bronze layer](https://github.com/jan-dyndor/life_etl/blob/main/ETL/Bronze_Layer_Autoloader.ipynb): `ETL/Bronze_Layer_Autoloader.ipynb`
- [Silver processing](https://github.com/jan-dyndor/life_etl/blob/main/ETL/Silver_Layer_Autoloader.ipynb): `ETL/Silver_Layer_Autoloader.ipynb`
- [Silver layer](https://github.com/jan-dyndor/life_etl/blob/main/ETL/Silver_Layer_Autoloader.ipynb): `ETL/Silver_Layer_Autoloader.ipynb`
- [Gold processing](https://github.com/jan-dyndor/life_etl/blob/main/ETL/Gold_Autoloader.ipynb): `ETL/Gold_Autoloader.ipynb`
- [Gold layer](https://github.com/jan-dyndor/life_etl/blob/main/ETL/Gold_Autoloader.ipynb): `ETL/Gold_Autoloader.ipynb`

### Streamlit Analytics

- [Project overview](https://github.com/jan-dyndor/life_etl/blob/main/app.py): `app.py`
- [Analytics page](https://github.com/jan-dyndor/life_etl/blob/main/pages/analytics_page.py): `pages/analytics_page.py`
- [Load and filter](https://github.com/jan-dyndor/life_etl/blob/main/streamlit_app/functions/functions.py): `streamlit_app/functions/functions.py`
- [Filtered FX data](https://github.com/jan-dyndor/life_etl/blob/main/streamlit_app/functions/functions.py): `streamlit_app/functions/functions.py`
- [Charts and analysis](https://github.com/jan-dyndor/life_etl/blob/main/pages/analytics_page.py): `pages/analytics_page.py`

### Other components

- **Daily scheduler**
- **Databricks job**
- **NBP API**
- **GitHub dataset**
- **Email notification**
- **Dashboard user**

## Connections

- Daily scheduler → Databricks job: triggers daily
- Databricks job → Job orchestration: runs job
- Job orchestration → NBP extraction: orchestrates
- NBP extraction → NBP API: fetches rates
- NBP extraction → Source data: writes raw data
- Bronze ingestion → Source data: reads raw data
- Bronze ingestion → Bronze layer: writes layer
- Bronze exploration → Bronze layer: explores
- Silver processing → Bronze layer: reads layer
- Silver processing → Silver layer: writes layer
- Gold processing → Silver layer: reads layer
- Gold processing → Gold layer: writes layer
- GitHub export → Gold layer: reads dataset
- GitHub export → GitHub dataset: exports dataset
- Databricks job → Email notification: notifies on success
- Dashboard user → Project overview: opens
- Dashboard user → Analytics page: selects page
- Analytics page → Load and filter: loads data
- Load and filter → GitHub dataset: requests CSV
- Load and filter → Filtered FX data: returns filtered data
- Analytics page → Charts and analysis: renders analysis
- Filtered FX data → Charts and analysis: feeds charts