How jpseveroo/projeto-cinedata-analytics works
CineData Analytics is a Databricks Lakehouse pipeline for turning five fragmented TMDB/IMDb CSVs and daily Bacen PTAX exchange rates into clean, standardized movie data, Gold analytics marts, and a movie-context table for RAG. Its principal workflow is Landing-to-Bronze ingestion, Bronze-to-Silver cleansing and typing, then Silver-to-Gold modeling. The README documents daily sequential job orchestration and downstream BI and AI consumers. Relationships here follow that documentation; notebook source excerpts were unavailable, so implementation-level transformations and finer-grained calls are not shown.
Main entry points: Silver_to_Gold.ipynb Silver to Gold · Landing_to_Bronze.ipynb Landing to Bronze · job.yaml Databricks Job · Bronze_to_Silver.ipynb Bronze to Silver
Components 12
Orchestration
- Databricks Job
job.yaml
Ingestion
- Landing to Bronze
Notebooks/Landing_to_Bronze.ipynb - Bronze tables
Data refinement
- Bronze to Silver
Notebooks/Bronze_to_Silver.ipynb - Silver tables
Analytics products
- Silver to Gold
Notebooks/Silver_to_Gold.ipynb - Gold star schema
- Movie context table
Other components
- TMDB / IMDb CSVs
Inputs - Atividade Engenharia de Dados - Bases de Dados - Bacen PTAX API
- BI consumers
- RAG assistant
Connections 13
- Databricks Job → Landing to Bronze: runs first
- Databricks Job → Bronze to Silver: runs second
- Databricks Job → Silver to Gold: runs third
- TMDB / IMDb CSVs → Landing to Bronze: provides CSVs
- Bacen PTAX API → Landing to Bronze: provides rates
- Landing to Bronze → Bronze tables: writes raw data
- Bronze tables → Bronze to Silver: reads raw data
- Bronze to Silver → Silver tables: writes cleaned data
- Silver tables → Silver to Gold: reads refined data
- Silver to Gold → Gold star schema: builds schema
- Silver to Gold → Movie context table: builds context
- Gold star schema → BI consumers: serves analytics
- Movie context table → RAG assistant: supports retrieval