How jan-dyndor/life_etl works
Life_ETL is a Databricks pipeline for daily NBP foreign-exchange data, with a Streamlit dashboard for inspecting the exported Gold dataset. The documented workflow runs from scheduled extraction through Bronze, Silver and Gold processing to GitHub export and notification; dashboard users load, filter and visualize the published data. Notebook names and the README support the ETL stages, but their internal code was not sampled, so their detailed transformations and wiring are not inferred. Dashboard behavior is supported by the sampled app and analytics sources.
Main entry points: download_data_from_npb_API.ipynb NBP extraction · analytics_page.py Analytics page · functions.py Load and filter · Bronze_Layer_Autoloader.ipynb Bronze layer
Components 22
Databricks ETL
- Job orchestration
ETL/pipeline_config.ipynb - NBP extraction
ETL/download_data_from_npb_API.ipynb - Bronze exploration
ETL/EDA_bronze.ipynb - GitHub export
ETL/Export_Data_Into_Volue_And_Send_To_GitHub.ipynb
Medallion Data
- Source data
ETL/download_data_from_npb_API.ipynb - Bronze ingestion
ETL/Bronze_Layer_Autoloader.ipynb - Bronze layer
ETL/Bronze_Layer_Autoloader.ipynb - Silver processing
ETL/Silver_Layer_Autoloader.ipynb - Silver layer
ETL/Silver_Layer_Autoloader.ipynb - Gold processing
ETL/Gold_Autoloader.ipynb - Gold layer
ETL/Gold_Autoloader.ipynb
Streamlit Analytics
- Project overview
app.py - Analytics page
pages/analytics_page.py - Load and filter
streamlit_app/functions/functions.py - Filtered FX data
streamlit_app/functions/functions.py - Charts and analysis
pages/analytics_page.py
Other components
- Daily scheduler
- Databricks job
- NBP API
- GitHub dataset
- Email notification
- Dashboard user
Connections 22
- Daily scheduler → Databricks job: triggers daily
- Databricks job → Job orchestration: runs job
- Job orchestration → NBP extraction: orchestrates
- NBP extraction → NBP API: fetches rates
- NBP extraction → Source data: writes raw data
- Bronze ingestion → Source data: reads raw data
- Bronze ingestion → Bronze layer: writes layer
- Bronze exploration → Bronze layer: explores
- Silver processing → Bronze layer: reads layer
- Silver processing → Silver layer: writes layer
- Gold processing → Silver layer: reads layer
- Gold processing → Gold layer: writes layer
- GitHub export → Gold layer: reads dataset
- GitHub export → GitHub dataset: exports dataset
- Databricks job → Email notification: notifies on success
- Dashboard user → Project overview: opens
- Dashboard user → Analytics page: selects page
- Analytics page → Load and filter: loads data
- Load and filter → GitHub dataset: requests CSV
- Load and filter → Filtered FX data: returns filtered data
- Analytics page → Charts and analysis: renders analysis
- Filtered FX data → Charts and analysis: feeds charts