How duyanhnguyen1902/olist-modern-data-platform works
Olist Modern Data Platform profiles and validates e-commerce CSVs, loads them into PostgreSQL, and builds tested analytics models for BI. Its full-refresh Airflow DAG sequences profiling, contract checks, database initialization, raw loading and reconciliation before invoking dbt. A separate incremental DAG checks watermark readiness, fans out source-to-RAW ingestion for six datasets, then builds dbt models. The README documents Metabase dashboards and read-only warehouse access. dbt model lineage is documented, but the sampled source does not show the model SQL dependencies; unsampled details are represented only where README supports them.
Main entry points: olist_full_refresh.py Full Refresh DAG · cli.py Pipeline CLI · incremental.py Watermark Ingestion · quality.py Contract Validation
Components 18
Ingestion
- OLTP Source
sql/06_create_source_tables.sql - Watermark Ingestion
src/olist_pipeline/incremental.py
Data Quality
- Source Profiling
src/olist_pipeline/profiling.py - Contract Validation
src/olist_pipeline/quality.py - Data Contracts
config/data_contracts.json
Warehouse and BI
- PostgreSQL
src/olist_pipeline/database.py - RAW Tables
sql/02_create_raw_tables.sql - Load Audit
src/olist_pipeline/incremental.py - Staging Models
dbt/models/staging - Kimball Models
dbt/models/marts - Analytics Marts
dbt/models/analytics - Metabase BI
Orchestration
- Full Refresh DAG
airflow/dags/olist_full_refresh.py - Incremental DAG
airflow/dags/olist_incremental.py
Pipeline Runtime
- dbt Runner
src/olist_pipeline/dbt_runner.py - Pipeline CLI
src/olist_pipeline/cli.py - Runtime Settings
src/olist_pipeline/settings.py
Other components
- Olist CSVs
data/raw
Connections 25
- Full Refresh DAG → Source Profiling: profiles
airflow/dags/olist_full_refresh.py - Full Refresh DAG → Contract Validation: validates
airflow/dags/olist_full_refresh.py - Full Refresh DAG → PostgreSQL: initializes
airflow/dags/olist_full_refresh.py - Full Refresh DAG → dbt Runner: builds via
airflow/dags/olist_full_refresh.py - Source Profiling → Olist CSVs: reads
src/olist_pipeline/profiling.py - Contract Validation → Olist CSVs: validates
src/olist_pipeline/quality.py - Contract Validation → Data Contracts: loads
src/olist_pipeline/quality.py - PostgreSQL → RAW Tables: loads
src/olist_pipeline/database.py - Incremental DAG → Watermark Ingestion: dispatches
airflow/dags/olist_incremental.py - Incremental DAG → dbt Runner: builds via
airflow/dags/olist_incremental.py - Watermark Ingestion → OLTP Source: extracts
src/olist_pipeline/incremental.py - Watermark Ingestion → RAW Tables: appends
src/olist_pipeline/incremental.py - Watermark Ingestion → Load Audit: updates
src/olist_pipeline/incremental.py - dbt Runner → Staging Models: runs dbt
src/olist_pipeline/dbt_runner.py - Staging Models → RAW Tables: reads sources
README.md - Staging Models → Kimball Models: transforms
README.md - Kimball Models → Analytics Marts: aggregates
README.md - Metabase BI → Analytics Marts: queries
README.md - Pipeline CLI → Source Profiling: invokes
src/olist_pipeline/cli.py - Pipeline CLI → Contract Validation: invokes
src/olist_pipeline/cli.py - Pipeline CLI → PostgreSQL: invokes
src/olist_pipeline/cli.py - Pipeline CLI → Watermark Ingestion: invokes
src/olist_pipeline/cli.py - Pipeline CLI → dbt Runner: invokes
src/olist_pipeline/cli.py - Full Refresh DAG → Runtime Settings: uses
airflow/dags/olist_full_refresh.py - Incremental DAG → Runtime Settings: uses
airflow/dags/olist_incremental.py