# rpalloni/airbyte-airflow-datalakehouse-dbt-metabase architecture

> Architecture diagram of the GitHub repository rpalloni/airbyte-airflow-datalakehouse-dbt-metabase, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase
- Repository: https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase

## Overview

This repository is a local analytics/lakehouse stack centered on Dremio, Nessie, MinIO, dbt, Airflow, Metabase, and an external Postgres source. It is not a single application; it is a containerized data platform with a transformation project and orchestration code.

High-level architecture

The platform has four main layers:

1. Source system
- A PostgreSQL database acts as the operational source of truth.
- The README defines sample domain tables: `users` and `bookings`.

2. Ingestion and storage
- Airbyte is used to extract data from Postgres.
- Data lands in MinIO, which serves as the S3-compatible object store.
- Nessie provides the table/catalog metadata layer for Iceberg-style table management.

3. Transformation and orchestration
- Dremio is the SQL/query engine and the place where dbt models are executed against the lakehouse.
- dbt, in the `transformer/` project, defines the transformation graph.
- Airflow orchestrates the dbt transformation flow through the DAG in `dags/dag_transformer.py`.

4. Consumption
- Metabase connects to Dremio for BI and reporting.
- A custom Metabase image includes a Dremio driver/plugin, so Metabase can query Dremio directly.

Main data flow

The intended end-to-end flow is:

Postgres source tables -> Airbyte sync -> MinIO storage + Nessie catalog -> Dremio query engine -> dbt transformations -> curated datasets for Metabase

A second control flow exists:

Airflow DAG -> runs/coordinates dbt transformation jobs -> updates curated Dremio datasets

The core idea is that raw operational data is ingested into a lakehouse, transformed in dbt, and exposed through Dremio to BI.

Repository boundaries

The repo is primarily split into three architectural areas:

- `docker/`
  - Compose files for the individual platform pieces:
    - `docker-compose-pg-source.yml`
    - `docker-compose-datalakehouse.yml`
    - `docker-compose-airflow.yml`
    - `docker-compose-metabase.yml`
    - `docker-compose-transformer.yml`
  - Custom Dockerfiles for the transformer and Metabase images.
  - This directory defines the runtime topology and service wiring.

- `dags/`
  - Contains Airflow DAG code.
  - `dag_transformer.py` is the orchestration entrypoint for the transformation workflow.

- `transformer/`
  - This is the dbt project.
  - `dbt_project.yml` and `profiles.yml` define how dbt connects to Dremio and where models are materialized.
  - Model layers are organized by purpose:
    - `models/staging/`
    - `models/marts/`
    - `models/reports/`
  - `models/sources.yml` defines raw source references.
  - `macros/`, `seeds/`, `snapshots/`, `tests/`, and `analyses/` are present as dbt extension points, though most are currently scaffolding.

dbt model hierarchy

The dbt project is clearly layered:

- Staging layer
  - `stg_users.sql`
  - `stg_bookings.sql`
  - This layer standardizes raw source data from Postgres/Airbyte.

- Mart layer
  - `dim_users.sql`
  - `fct_bookings.sql`
  - This layer creates dimensional and fact structures suitable for analytics.

- Reporting layer
  - `rpt_user_stats.sql`
  - This is the BI-facing output, likely joining marts into a consumable summary dataset.

This layering is important for architecture diagrams because it shows the progression from raw data to modeled analytics assets.

Dremio / Nessie / MinIO relationship

The README describes a catalog-first lakehouse setup:

- MinIO stores the physical data files.
- Nessie tracks versioned table/catalog metadata.
- Dremio reads from both and exposes a SQL interface.
- dbt targets Dremio, and depending on the materialization, Dremio writes either logical views or physical Iceberg tables backed by MinIO.

This distinction matters:
- Dremio Space is the logical namespace.
- Nessie catalog is the versioned namespace when working catalog-first.
- Object storage source/path determine where physical table data lives in MinIO.

So in a diagram, Nessie should sit between Dremio and MinIO as the metadata/catalog boundary.

External interfaces and ports

The README exposes the main user-facing endpoints:
- Airbyte UI: `localhost:8000`
- Airflow UI: `localhost:8081`
- Metabase UI: `localhost:3000`
- MinIO console: `localhost:9001`
- Dremio UI: `localhost:9047`
- Nessie API: `localhost:19120`

There is also a Dremio query endpoint used by Metabase via the custom plugin:
- `dremio:31010`

Relevant tooling/runtime signals

- Python project managed with `pyproject.toml` and `.python-version`
- dbt adapter: `dbt-dremio`
- Local execution mentions `uv run dbt run`, so `uv` is the Python package/runtime workflow
- Airflow is configured via `config/airflow.cfg`
- Docker Compose is the primary deployment mechanism for local development

What to emphasize in an architecture diagram

If you are drawing the architecture, the most important boundaries are:

- Operational source boundary: Postgres
- Ingestion boundary: Airbyte
- Storage/catalog boundary: MinIO + Nessie
- Query/compute boundary: Dremio
- Transformation boundary: dbt project in `transformer/`
- Scheduling boundary: Airflow DAGs in `dags/`
- Consumption boundary: Metabase

The repo itself mostly contains the orchestration and transformation layers, plus the Docker wiring to stand up the local lakehouse stack. It is not a general-purpose app backend; it is a data platform repo focused on ELT/analytics.

Main entry points: [`docker/docker-compose-datalakehouse.yml`](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/docker/docker-compose-datalakehouse.yml) (Compose stack), [`transformer/dbt_project.yml`](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/transformer/dbt_project.yml) (dbt project), [`docker/docker-compose-airflow.yml`](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/docker/docker-compose-airflow.yml) (Airflow), [`dags/dag_transformer.py`](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/dags/dag_transformer.py) (Transformer DAG)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph "Source"
  postgres[("Postgres<br/>operational db")]
end

subgraph "Lakehouse"
  airbyte["Airbyte<br/>ingestion"]
  minio[("MinIO<br/>object store")]
  nessie["Nessie<br/>catalog"]
  dremio["Dremio<br/>query engine"]
end

subgraph "Orchestration"
  airflow["Airflow<br/>scheduler"]
  dag_transformer["Transformer DAG<br/>airflow dag<br/>[dag_transformer.py]"]
end

subgraph "Transform"
  dbt_project["dbt project<br/>transformation project<br/>[dbt_project.yml]"]
  profiles["Profiles<br/>dbt config<br/>[profiles.yml]"]
  sources_yml["Sources<br/>[sources.yml]"]
  staging["Staging<br/>dbt models<br/>[stg_users.sql]"]
  marts["Marts<br/>dbt models<br/>[dim_users.sql]"]
  reports["Reports<br/>dbt models<br/>[rpt_user_stats.sql]"]
end

subgraph "Consumption"
  metabase["Metabase<br/>bi tool"]
end

subgraph "Runtime wiring"
  docker_stack["Compose stack<br/>docker wiring"]
  transformer_image["Transformer image<br/>custom image"]
  metabase_image["Metabase image<br/>custom image"]
end

postgres -->|"extract"| airbyte
airbyte -->|"load"| minio
minio -.->|"catalogs"| nessie
nessie -->|"metadata"| dremio
minio -->|"reads"| dremio
airflow -->|"runs"| dag_transformer
dag_transformer -->|"orchestrates"| dbt_project
dbt_project -.->|"configures"| profiles
dbt_project -.->|"declares"| sources_yml
sources_yml -->|"feeds"| staging
staging -->|"builds"| marts
marts -->|"summarizes"| reports
dbt_project -->|"targets"| dremio
reports -->|"publishes"| dremio
metabase -->|"queries"| dremio
docker_stack -.->|"wires"| airbyte
docker_stack -.->|"wires"| minio
docker_stack -.->|"wires"| nessie
docker_stack -.->|"wires"| dremio
docker_stack -.->|"wires"| airflow
docker_stack -.->|"wires"| metabase
docker_stack -.->|"builds"| transformer_image
docker_stack -.->|"builds"| metabase_image

click airbyte "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/docker/docker-compose-datalakehouse.yml"
click minio "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/docker/docker-compose-datalakehouse.yml"
click nessie "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/docker/docker-compose-datalakehouse.yml"
click dremio "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/docker/docker-compose-datalakehouse.yml"
click airflow "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/docker/docker-compose-airflow.yml"
click dag_transformer "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/dags/dag_transformer.py"
click dbt_project "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/transformer/dbt_project.yml"
click profiles "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/transformer/profiles.yml"
click sources_yml "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/transformer/models/sources.yml"
click staging "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/transformer/models/staging/stg_users.sql"
click marts "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/transformer/models/marts/dim_users.sql"
click reports "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/transformer/models/reports/rpt_user_stats.sql"
click metabase "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/docker/docker-compose-metabase.yml"
click docker_stack "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/docker/docker-compose-datalakehouse.yml"
click transformer_image "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/docker/Dockerfile.transformer"
click metabase_image "https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/main/docker/Dockerfile.metabase"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class postgres toneBlue
class airbyte,minio,nessie,dremio toneAmber
class airflow,dag_transformer toneMint
class dbt_project,profiles,sources_yml,staging,marts,reports toneRose
class metabase toneIndigo
class docker_stack,transformer_image,metabase_image toneTeal
```

## Components

### Source

Operational source system

- **Postgres**: operational db · External source of truth with users and bookings data

### Lakehouse

Storage, catalog, and query engine

- [Airbyte](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/docker/docker-compose-datalakehouse.yml): ingestion · Extracts source data into the lakehouse · `docker/docker-compose-datalakehouse.yml`
- [MinIO](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/docker/docker-compose-datalakehouse.yml): object store · S3-compatible physical storage for lakehouse data · `docker/docker-compose-datalakehouse.yml`
- [Nessie](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/docker/docker-compose-datalakehouse.yml): catalog · Versioned table and metadata catalog boundary · `docker/docker-compose-datalakehouse.yml`
- [Dremio](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/docker/docker-compose-datalakehouse.yml): query engine · SQL execution layer for lakehouse tables and dbt targets · `docker/docker-compose-datalakehouse.yml`

### Orchestration

Scheduling and workflow control

- [Airflow](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/docker/docker-compose-airflow.yml): scheduler · Orchestrates transformation workflows · `docker/docker-compose-airflow.yml`
- [Transformer DAG](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/dags/dag_transformer.py): airflow dag · Coordinates dbt transformation runs · `dags/dag_transformer.py`

### Transform

dbt project and model layers

- [dbt project](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/transformer/dbt_project.yml): transformation project · dbt configuration and model graph targeting Dremio · `transformer/dbt_project.yml`
- [Profiles](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/transformer/profiles.yml): dbt config · dbt connection settings for Dremio · `transformer/profiles.yml`
- [Sources](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/transformer/models/sources.yml): dbt sources · Raw source declarations for staged models · `transformer/models/sources.yml`
- [Staging](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/transformer/models/staging/stg_users.sql): dbt models · Normalizes raw Postgres/Airbyte data · `transformer/models/staging/stg_users.sql`
- [Marts](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/transformer/models/marts/dim_users.sql): dbt models · Analytic dimensions and facts · `transformer/models/marts/dim_users.sql`
- [Reports](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/transformer/models/reports/rpt_user_stats.sql): dbt models · BI-facing summary datasets · `transformer/models/reports/rpt_user_stats.sql`

### Consumption

BI and reporting layer

- [Metabase](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/docker/docker-compose-metabase.yml): bi tool · Reporting and exploration layer querying Dremio · `docker/docker-compose-metabase.yml`

### Runtime wiring

Docker Compose and custom images

- [Compose stack](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/docker/docker-compose-datalakehouse.yml): docker wiring · Local runtime topology for platform services · `docker/docker-compose-datalakehouse.yml`
- [Transformer image](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/docker/Dockerfile.transformer): custom image · Custom dbt runtime container · `docker/Dockerfile.transformer`
- [Metabase image](https://github.com/rpalloni/airbyte-airflow-datalakehouse-dbt-metabase/blob/HEAD/docker/Dockerfile.metabase): custom image · Custom Metabase image with Dremio driver support · `docker/Dockerfile.metabase`

## Connections

- Postgres → Airbyte: extract · Operational data is pulled from Postgres
- Airbyte → MinIO: load · Synced data lands in object storage
- MinIO → Nessie: catalogs · Nessie tracks versioned lakehouse metadata over stored data
- Nessie → Dremio: metadata · Dremio resolves tables through the catalog boundary
- MinIO → Dremio: reads · Dremio queries physical lakehouse files
- Airflow → Transformer DAG: runs · Airflow loads and schedules the DAG
- Transformer DAG → dbt project: orchestrates · The DAG triggers dbt transformation execution
- dbt project → Profiles: configures · Project uses Dremio connection settings
- dbt project → Sources: declares · Defines raw source dependencies
- Sources → Staging: feeds · Staging models standardize raw source data
- Staging → Marts: builds · Staging outputs flow into dimensional and fact models
- Marts → Reports: summarizes · Marts are rolled into BI-ready reporting datasets
- dbt project → Dremio: targets · dbt executes transformations against Dremio
- Reports → Dremio: publishes · Curated datasets are exposed through Dremio
- Metabase → Dremio: queries · BI connects directly to Dremio for analytics
- Compose stack → Airbyte: wires · Compose files define local service topology
- Compose stack → MinIO: wires · Compose files define local service topology
- Compose stack → Nessie: wires · Compose files define local service topology
- Compose stack → Dremio: wires · Compose files define local service topology
- Compose stack → Airflow: wires · Compose files define local service topology
- Compose stack → Metabase: wires · Compose files define local service topology
- Compose stack → Transformer image: builds · Custom dbt runtime image is part of the stack
- Compose stack → Metabase image: builds · Custom Metabase image is part of the stack
