# duyanhnguyen1902/olist-modern-data-platform architecture

> Architecture diagram of the GitHub repository duyanhnguyen1902/olist-modern-data-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/duyanhnguyen1902/olist-modern-data-platform
- Repository: https://github.com/duyanhnguyen1902/olist-modern-data-platform

## Overview

Olist Modern Data Platform profiles and validates e-commerce CSVs, loads them into PostgreSQL, and builds tested analytics models for BI. Its full-refresh Airflow DAG sequences profiling, contract checks, database initialization, raw loading and reconciliation before invoking dbt. A separate incremental DAG checks watermark readiness, fans out source-to-RAW ingestion for six datasets, then builds dbt models. The README documents Metabase dashboards and read-only warehouse access. dbt model lineage is documented, but the sampled source does not show the model SQL dependencies; unsampled details are represented only where README supports them.

Main entry points: [`airflow/dags/olist_full_refresh.py`](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/airflow/dags/olist_full_refresh.py) (Full Refresh DAG), [`src/olist_pipeline/cli.py`](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/cli.py) (Pipeline CLI), [`src/olist_pipeline/incremental.py`](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/incremental.py) (Watermark Ingestion), [`src/olist_pipeline/quality.py`](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/quality.py) (Contract Validation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion"]
  node_source_db[("OLTP Source")]
  node_incremental["Watermark Ingestion<br/>[incremental.py]"]
end

subgraph group_quality["Data Quality"]
  node_profile["Source Profiling<br/>[profiling.py]"]
  node_contracts["Contract Validation<br/>[quality.py]"]
  node_contract_config["Data Contracts"]
end

subgraph group_warehouse["Warehouse and BI"]
  node_postgres[("PostgreSQL<br/>[database.py]")]
  node_raw[("RAW Tables")]
  node_audit[("Load Audit<br/>[incremental.py]")]
  node_staging["Staging Models"]
  node_kimball["Kimball Models"]
  node_analytics["Analytics Marts"]
  node_metabase["Metabase BI"]
end

subgraph group_orchestration["Orchestration"]
  node_full_dag["Full Refresh DAG"]
  node_incremental_dag["Incremental DAG"]
end

subgraph group_runtime["Pipeline Runtime"]
  node_dbt_runner["dbt Runner<br/>[dbt_runner.py]"]
  node_cli["Pipeline CLI<br/>[cli.py]"]
  node_settings["Runtime Settings<br/>[settings.py]"]
end

node_csv["Olist CSVs"]

node_full_dag -->|"profiles"| node_profile
node_full_dag -->|"validates"| node_contracts
node_full_dag -->|"initializes"| node_postgres
node_full_dag -->|"builds via"| node_dbt_runner
node_profile -->|"reads"| node_csv
node_contracts -->|"validates"| node_csv
node_contracts -->|"loads"| node_contract_config
node_postgres -->|"loads"| node_raw
node_incremental_dag -->|"dispatches"| node_incremental
node_incremental_dag -->|"builds via"| node_dbt_runner
node_incremental -->|"extracts"| node_source_db
node_incremental -->|"appends"| node_raw
node_incremental -->|"updates"| node_audit
node_dbt_runner -->|"runs dbt"| node_staging
node_staging -.->|"reads sources"| node_raw
node_staging -.->|"transforms"| node_kimball
node_kimball -.->|"aggregates"| node_analytics
node_metabase -.->|"queries"| node_analytics
node_cli -->|"invokes"| node_profile
node_cli -->|"invokes"| node_contracts
node_cli -->|"invokes"| node_postgres
node_cli -->|"invokes"| node_incremental
node_cli -->|"invokes"| node_dbt_runner
node_full_dag -->|"uses"| node_settings
node_incremental_dag -->|"uses"| node_settings

click node_csv "https://github.com/duyanhnguyen1902/olist-modern-data-platform/tree/main/data/raw"
click node_profile "https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/profiling.py"
click node_contracts "https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/quality.py"
click node_contract_config "https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/config/data_contracts.json"
click node_full_dag "https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/airflow/dags/olist_full_refresh.py"
click node_incremental_dag "https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/airflow/dags/olist_incremental.py"
click node_source_db "https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/sql/06_create_source_tables.sql"
click node_incremental "https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/incremental.py"
click node_postgres "https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/database.py"
click node_raw "https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/sql/02_create_raw_tables.sql"
click node_audit "https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/incremental.py"
click node_dbt_runner "https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/dbt_runner.py"
click node_staging "https://github.com/duyanhnguyen1902/olist-modern-data-platform/tree/main/dbt/models/staging"
click node_kimball "https://github.com/duyanhnguyen1902/olist-modern-data-platform/tree/main/dbt/models/marts"
click node_analytics "https://github.com/duyanhnguyen1902/olist-modern-data-platform/tree/main/dbt/models/analytics"
click node_cli "https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/cli.py"
click node_settings "https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/settings.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_source_db,node_incremental toneBlue
class node_profile,node_contracts,node_contract_config toneAmber
class node_postgres,node_raw,node_audit,node_staging,node_kimball,node_analytics,node_metabase toneMint
class node_full_dag,node_incremental_dag toneRose
class node_dbt_runner,node_cli,node_settings toneIndigo
class node_csv toneTeal
```

## Components

### Ingestion

- [OLTP Source](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/sql/06_create_source_tables.sql): `sql/06_create_source_tables.sql`
- [Watermark Ingestion](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/incremental.py): `src/olist_pipeline/incremental.py`

### Data Quality

- [Source Profiling](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/profiling.py): `src/olist_pipeline/profiling.py`
- [Contract Validation](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/quality.py): `src/olist_pipeline/quality.py`
- [Data Contracts](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/config/data_contracts.json): `config/data_contracts.json`

### Warehouse and BI

- [PostgreSQL](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/database.py): `src/olist_pipeline/database.py`
- [RAW Tables](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/sql/02_create_raw_tables.sql): `sql/02_create_raw_tables.sql`
- [Load Audit](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/incremental.py): `src/olist_pipeline/incremental.py`
- [Staging Models](https://github.com/duyanhnguyen1902/olist-modern-data-platform/tree/main/dbt/models/staging): `dbt/models/staging`
- [Kimball Models](https://github.com/duyanhnguyen1902/olist-modern-data-platform/tree/main/dbt/models/marts): `dbt/models/marts`
- [Analytics Marts](https://github.com/duyanhnguyen1902/olist-modern-data-platform/tree/main/dbt/models/analytics): `dbt/models/analytics`
- **Metabase BI**

### Orchestration

- [Full Refresh DAG](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/airflow/dags/olist_full_refresh.py): `airflow/dags/olist_full_refresh.py`
- [Incremental DAG](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/airflow/dags/olist_incremental.py): `airflow/dags/olist_incremental.py`

### Pipeline Runtime

- [dbt Runner](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/dbt_runner.py): `src/olist_pipeline/dbt_runner.py`
- [Pipeline CLI](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/cli.py): `src/olist_pipeline/cli.py`
- [Runtime Settings](https://github.com/duyanhnguyen1902/olist-modern-data-platform/blob/main/src/olist_pipeline/settings.py): `src/olist_pipeline/settings.py`

### Other components

- [Olist CSVs](https://github.com/duyanhnguyen1902/olist-modern-data-platform/tree/main/data/raw): `data/raw`

## Connections

- Full Refresh DAG → Source Profiling: profiles (evidence: `airflow/dags/olist_full_refresh.py`)
- Full Refresh DAG → Contract Validation: validates (evidence: `airflow/dags/olist_full_refresh.py`)
- Full Refresh DAG → PostgreSQL: initializes (evidence: `airflow/dags/olist_full_refresh.py`)
- Full Refresh DAG → dbt Runner: builds via (evidence: `airflow/dags/olist_full_refresh.py`)
- Source Profiling → Olist CSVs: reads (evidence: `src/olist_pipeline/profiling.py`)
- Contract Validation → Olist CSVs: validates (evidence: `src/olist_pipeline/quality.py`)
- Contract Validation → Data Contracts: loads (evidence: `src/olist_pipeline/quality.py`)
- PostgreSQL → RAW Tables: loads (evidence: `src/olist_pipeline/database.py`)
- Incremental DAG → Watermark Ingestion: dispatches (evidence: `airflow/dags/olist_incremental.py`)
- Incremental DAG → dbt Runner: builds via (evidence: `airflow/dags/olist_incremental.py`)
- Watermark Ingestion → OLTP Source: extracts (evidence: `src/olist_pipeline/incremental.py`)
- Watermark Ingestion → RAW Tables: appends (evidence: `src/olist_pipeline/incremental.py`)
- Watermark Ingestion → Load Audit: updates (evidence: `src/olist_pipeline/incremental.py`)
- dbt Runner → Staging Models: runs dbt (evidence: `src/olist_pipeline/dbt_runner.py`)
- Staging Models → RAW Tables: reads sources (evidence: `README.md`)
- Staging Models → Kimball Models: transforms (evidence: `README.md`)
- Kimball Models → Analytics Marts: aggregates (evidence: `README.md`)
- Metabase BI → Analytics Marts: queries (evidence: `README.md`)
- Pipeline CLI → Source Profiling: invokes (evidence: `src/olist_pipeline/cli.py`)
- Pipeline CLI → Contract Validation: invokes (evidence: `src/olist_pipeline/cli.py`)
- Pipeline CLI → PostgreSQL: invokes (evidence: `src/olist_pipeline/cli.py`)
- Pipeline CLI → Watermark Ingestion: invokes (evidence: `src/olist_pipeline/cli.py`)
- Pipeline CLI → dbt Runner: invokes (evidence: `src/olist_pipeline/cli.py`)
- Full Refresh DAG → Runtime Settings: uses (evidence: `airflow/dags/olist_full_refresh.py`)
- Incremental DAG → Runtime Settings: uses (evidence: `airflow/dags/olist_incremental.py`)
