How rpalloni/airbyte-airflow-datalakehouse-dbt-metabase works
This repository is a local analytics/lakehouse stack centered on Dremio, Nessie, MinIO, dbt, Airflow, Metabase, and an external Postgres source. It is not a single application; it is a containerized data platform with a transformation project and orchestration code.
High-level architecture
The platform has four main layers:
1. Source system
• A PostgreSQL database acts as the operational source of truth.
• The README defines sample domain tables: users and bookings.
2. Ingestion and storage
• Airbyte is used to extract data from Postgres.
• Data lands in MinIO, which serves as the S3-compatible object store.
• Nessie provides the table/catalog metadata layer for Iceberg-style table management.
3. Transformation and orchestration
• Dremio is the SQL/query engine and the place where dbt models are executed against the lakehouse.
• dbt, in the transformer/ project, defines the transformation graph.
• Airflow orchestrates the dbt transformation flow through the DAG in dags/dag_transformer.py.
4. Consumption
• Metabase connects to Dremio for BI and reporting.
• A custom Metabase image includes a Dremio driver/plugin, so Metabase can query Dremio directly.
Main data flow
The intended end-to-end flow is:
Postgres source tables -> Airbyte sync -> MinIO storage + Nessie catalog -> Dremio query engine -> dbt transformations -> curated datasets for Metabase
A second control flow exists:
Airflow DAG -> runs/coordinates dbt transformation jobs -> updates curated Dremio datasets
The core idea is that raw operational data is ingested into a lakehouse, transformed in dbt, and exposed through Dremio to BI.
Repository boundaries
The repo is primarily split into three architectural areas:
• docker/
- Compose files for the individual platform pieces:
- docker-compose-pg-source.yml
- docker-compose-datalakehouse.yml
- docker-compose-airflow.yml
- docker-compose-metabase.yml
- docker-compose-transformer.yml
- Custom Dockerfiles for the transformer and Metabase images.
- This directory defines the runtime topology and service wiring.
• dags/
- Contains Airflow DAG code.
- dag_transformer.py is the orchestration entrypoint for the transformation workflow.
• transformer/
- This is the dbt project.
- dbt_project.yml and profiles.yml define how dbt connects to Dremio and where models are materialized.
- Model layers are organized by purpose:
- models/staging/
- models/marts/
- models/reports/
- models/sources.yml defines raw source references.
- macros/, seeds/, snapshots/, tests/, and analyses/ are present as dbt extension points, though most are currently scaffolding.
dbt model hierarchy
The dbt project is clearly layered:
• Staging layer
- stg_users.sql
- stg_bookings.sql
- This layer standardizes raw source data from Postgres/Airbyte.
• Mart layer
- dim_users.sql
- fct_bookings.sql
- This layer creates dimensional and fact structures suitable for analytics.
• Reporting layer
- rpt_user_stats.sql
- This is the BI-facing output, likely joining marts into a consumable summary dataset.
This layering is important for architecture diagrams because it shows the progression from raw data to modeled analytics assets.
Dremio / Nessie / MinIO relationship
The README describes a catalog-first lakehouse setup:
• MinIO stores the physical data files.
• Nessie tracks versioned table/catalog metadata.
• Dremio reads from both and exposes a SQL interface.
• dbt targets Dremio, and depending on the materialization, Dremio writes either logical views or physical Iceberg tables backed by MinIO.
This distinction matters:
• Dremio Space is the logical namespace.
• Nessie catalog is the versioned namespace when working catalog-first.
• Object storage source/path determine where physical table data lives in MinIO.
So in a diagram, Nessie should sit between Dremio and MinIO as the metadata/catalog boundary.
External interfaces and ports
The README exposes the main user-facing endpoints:
• Airbyte UI: localhost:8000
• Airflow UI: localhost:8081
• Metabase UI: localhost:3000
• MinIO console: localhost:9001
• Dremio UI: localhost:9047
• Nessie API: localhost:19120
There is also a Dremio query endpoint used by Metabase via the custom plugin:
• dremio:31010
Relevant tooling/runtime signals
• Python project managed with pyproject.toml and .python-version
• dbt adapter: dbt-dremio
• Local execution mentions uv run dbt run, so uv is the Python package/runtime workflow
• Airflow is configured via config/airflow.cfg
• Docker Compose is the primary deployment mechanism for local development
What to emphasize in an architecture diagram
If you are drawing the architecture, the most important boundaries are:
• Operational source boundary: Postgres
• Ingestion boundary: Airbyte
• Storage/catalog boundary: MinIO + Nessie
• Query/compute boundary: Dremio
• Transformation boundary: dbt project in transformer/
• Scheduling boundary: Airflow DAGs in dags/
• Consumption boundary: Metabase
The repo itself mostly contains the orchestration and transformation layers, plus the Docker wiring to stand up the local lakehouse stack. It is not a general-purpose app backend; it is a data platform repo focused on ELT/analytics.
Main entry points: docker-compose-datalakehouse.yml Compose stack · dbt_project.yml dbt project · docker-compose-airflow.yml Airflow · dag_transformer.py Transformer DAG
Components 17
Source
Operational source system
- Postgres · External source of truth with users and bookings data
Lakehouse
Storage, catalog, and query engine
- Airbyte · Extracts source data into the lakehouse
docker/docker-compose-datalakehouse.yml - MinIO · S3-compatible physical storage for lakehouse data
docker/docker-compose-datalakehouse.yml - Nessie · Versioned table and metadata catalog boundary
docker/docker-compose-datalakehouse.yml - Dremio · SQL execution layer for lakehouse tables and dbt targets
docker/docker-compose-datalakehouse.yml
Orchestration
Scheduling and workflow control
- Airflow · Orchestrates transformation workflows
docker/docker-compose-airflow.yml - Transformer DAG · Coordinates dbt transformation runs
dags/dag_transformer.py
Transform
dbt project and model layers
- dbt project · dbt configuration and model graph targeting Dremio
transformer/dbt_project.yml - Profiles · dbt connection settings for Dremio
transformer/profiles.yml - Sources · Raw source declarations for staged models
transformer/models/sources.yml - Staging · Normalizes raw Postgres/Airbyte data
transformer/models/staging/stg_users.sql - Marts · Analytic dimensions and facts
transformer/models/marts/dim_users.sql - Reports · BI-facing summary datasets
transformer/models/reports/rpt_user_stats.sql
Consumption
BI and reporting layer
- Metabase · Reporting and exploration layer querying Dremio
docker/docker-compose-metabase.yml
Runtime wiring
Docker Compose and custom images
- Compose stack · Local runtime topology for platform services
docker/docker-compose-datalakehouse.yml - Transformer image · Custom dbt runtime container
docker/Dockerfile.transformer - Metabase image · Custom Metabase image with Dremio driver support
docker/Dockerfile.metabase
Connections 23
- Postgres → Airbyte: extract · Operational data is pulled from Postgres
- Airbyte → MinIO: load · Synced data lands in object storage
- MinIO → Nessie: catalogs · Nessie tracks versioned lakehouse metadata over stored data
- Nessie → Dremio: metadata · Dremio resolves tables through the catalog boundary
- MinIO → Dremio: reads · Dremio queries physical lakehouse files
- Airflow → Transformer DAG: runs · Airflow loads and schedules the DAG
- Transformer DAG → dbt project: orchestrates · The DAG triggers dbt transformation execution
- dbt project → Profiles: configures · Project uses Dremio connection settings
- dbt project → Sources: declares · Defines raw source dependencies
- Sources → Staging: feeds · Staging models standardize raw source data
- Staging → Marts: builds · Staging outputs flow into dimensional and fact models
- Marts → Reports: summarizes · Marts are rolled into BI-ready reporting datasets
- dbt project → Dremio: targets · dbt executes transformations against Dremio
- Reports → Dremio: publishes · Curated datasets are exposed through Dremio
- Metabase → Dremio: queries · BI connects directly to Dremio for analytics
- Compose stack → Airbyte: wires · Compose files define local service topology
- Compose stack → MinIO: wires · Compose files define local service topology
- Compose stack → Nessie: wires · Compose files define local service topology
- Compose stack → Dremio: wires · Compose files define local service topology
- Compose stack → Airflow: wires · Compose files define local service topology
- Compose stack → Metabase: wires · Compose files define local service topology
- Compose stack → Transformer image: builds · Custom dbt runtime image is part of the stack
- Compose stack → Metabase image: builds · Custom Metabase image is part of the stack