# duckdb/ducklake architecture

> Architecture diagram of the GitHub repository duckdb/ducklake, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/duckdb/ducklake
- Repository: https://github.com/duckdb/ducklake

## Overview

DuckLake is a DuckDB extension that presents a SQL-accessible lakehouse: catalog metadata lives in a database, while table data lives in Parquet files. The documented workflow is attach a catalog, create or modify tables, then query them; snapshots enable time travel and change-feed queries. The graph shows the principal storage, transaction, and metadata layers plus documented maintenance and metadata-query capabilities. Several implementation relationships are inferred only from the SOURCE INDEX reference lists, so unsampled subsystems and uncertain wiring are omitted.

Main entry points: [`src/storage/ducklake_metadata_manager.cpp`](https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_metadata_manager.cpp) (Metadata manager), [`src/storage/ducklake_catalog.cpp`](https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_catalog.cpp) (DuckLake catalog), [`src/storage/ducklake_transaction.cpp`](https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_transaction.cpp) (Transactions), [`src/storage/ducklake_table_entry.cpp`](https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_table_entry.cpp) (Table entries)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sql["SQL integration"]
  node_extension["DuckLake extension"]
  node_sql_functions["SQL functions"]
  node_table_changes["Change feed"]
end

subgraph group_catalog["Catalog and transactions"]
  node_catalog["DuckLake catalog"]
  node_transactions["Transactions"]
  node_table_entry["Table entries"]
end

subgraph group_data["Data operations"]
  node_insert["Insert and writes"]
  node_add_files["Add data files"]
  node_scan["Table scanning<br/>[ducklake_scan.cpp]"]
  node_file_reader["Multi-file reader"]
  node_partition["Partitioning"]
  node_stats["File statistics<br/>[ducklake_stats.cpp]"]
  node_compaction["Compaction"]
end

subgraph group_metadata["Metadata and history"]
  node_metadata_manager["Metadata manager"]
  node_sqlite["SQLite catalog"]
  node_postgres["PostgreSQL catalog"]
  node_quack["Quack catalog"]
  node_snapshot["Snapshot history"]
  node_schema["Schema metadata"]
end

node_user(("SQL user"))
node_parquet[("Parquet files")]
node_catalog_db[("Catalog database")]

node_user -->|"attaches"| node_extension
node_extension -->|"registers storage"| node_catalog
node_extension -->|"registers functions"| node_sql_functions
node_sql_functions -.->|"accesses catalog"| node_catalog
node_catalog -->|"uses transactions"| node_transactions
node_transactions -->|"records changes"| node_metadata_manager
node_catalog -.->|"loads metadata"| node_metadata_manager
node_metadata_manager -->|"dispatches to"| node_sqlite
node_metadata_manager -->|"dispatches to"| node_postgres
node_metadata_manager -->|"dispatches to"| node_quack
node_metadata_manager -->|"reads and writes"| node_catalog_db
node_catalog -.->|"manages entries"| node_table_entry
node_transactions -->|"tracks snapshots"| node_snapshot
node_metadata_manager -->|"persists schemas"| node_schema
node_user -->|"queries changes"| node_table_changes
node_table_changes -.->|"reads history"| node_snapshot
node_user -->|"writes tables"| node_insert
node_insert -->|"writes files"| node_parquet
node_insert -->|"records files"| node_transactions
node_add_files -->|"updates changes"| node_transactions
node_user -->|"queries tables"| node_scan
node_scan -.->|"reads files"| node_file_reader
node_file_reader -->|"reads data"| node_parquet
node_scan -.->|"scans table"| node_table_entry
node_table_entry -->|"uses partition spec"| node_partition
node_table_entry -->|"provides statistics"| node_stats
node_user -->|"requests maintenance"| node_compaction
node_compaction -->|"reads files"| node_file_reader
node_compaction -->|"rewrites files"| node_insert

click node_extension "https://github.com/duckdb/ducklake/blob/main/src/ducklake_extension.cpp"
click node_sql_functions "https://github.com/duckdb/ducklake/tree/main/src/functions"
click node_catalog "https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_catalog.cpp"
click node_transactions "https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_transaction.cpp"
click node_table_entry "https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_table_entry.cpp"
click node_metadata_manager "https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_metadata_manager.cpp"
click node_sqlite "https://github.com/duckdb/ducklake/blob/main/src/metadata_manager/sqlite_metadata_manager.cpp"
click node_postgres "https://github.com/duckdb/ducklake/blob/main/src/metadata_manager/postgres_metadata_manager.cpp"
click node_quack "https://github.com/duckdb/ducklake/blob/main/src/metadata_manager/quack_metadata_manager.cpp"
click node_snapshot "https://github.com/duckdb/ducklake/blob/main/src/common/ducklake_snapshot.cpp"
click node_schema "https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_schema_entry.cpp"
click node_table_changes "https://github.com/duckdb/ducklake/blob/main/src/functions/ducklake_table_changes.cpp"
click node_insert "https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_insert.cpp"
click node_add_files "https://github.com/duckdb/ducklake/blob/main/src/functions/ducklake_add_data_files.cpp"
click node_scan "https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_scan.cpp"
click node_file_reader "https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_multi_file_reader.cpp"
click node_partition "https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_partition_data.cpp"
click node_stats "https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_stats.cpp"
click node_compaction "https://github.com/duckdb/ducklake/blob/main/src/functions/ducklake_compaction_functions.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_extension,node_sql_functions,node_table_changes,node_user toneBlue
class node_catalog,node_transactions,node_table_entry,node_parquet,node_catalog_db toneAmber
class node_insert,node_add_files,node_scan,node_file_reader,node_partition,node_stats,node_compaction toneMint
class node_metadata_manager,node_sqlite,node_postgres,node_quack,node_snapshot,node_schema toneRose
```

## Components

### SQL integration

- [DuckLake extension](https://github.com/duckdb/ducklake/blob/main/src/ducklake_extension.cpp): `src/ducklake_extension.cpp`
- [SQL functions](https://github.com/duckdb/ducklake/tree/main/src/functions): `src/functions`
- [Change feed](https://github.com/duckdb/ducklake/blob/main/src/functions/ducklake_table_changes.cpp): `src/functions/ducklake_table_changes.cpp`

### Catalog and transactions

- [DuckLake catalog](https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_catalog.cpp): `src/storage/ducklake_catalog.cpp`
- [Transactions](https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_transaction.cpp): `src/storage/ducklake_transaction.cpp`
- [Table entries](https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_table_entry.cpp): `src/storage/ducklake_table_entry.cpp`

### Data operations

- [Insert and writes](https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_insert.cpp): `src/storage/ducklake_insert.cpp`
- [Add data files](https://github.com/duckdb/ducklake/blob/main/src/functions/ducklake_add_data_files.cpp): `src/functions/ducklake_add_data_files.cpp`
- [Table scanning](https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_scan.cpp): `src/storage/ducklake_scan.cpp`
- [Multi-file reader](https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_multi_file_reader.cpp): `src/storage/ducklake_multi_file_reader.cpp`
- [Partitioning](https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_partition_data.cpp): `src/storage/ducklake_partition_data.cpp`
- [File statistics](https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_stats.cpp): `src/storage/ducklake_stats.cpp`
- [Compaction](https://github.com/duckdb/ducklake/blob/main/src/functions/ducklake_compaction_functions.cpp): `src/functions/ducklake_compaction_functions.cpp`

### Metadata and history

- [Metadata manager](https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_metadata_manager.cpp): `src/storage/ducklake_metadata_manager.cpp`
- [SQLite catalog](https://github.com/duckdb/ducklake/blob/main/src/metadata_manager/sqlite_metadata_manager.cpp): `src/metadata_manager/sqlite_metadata_manager.cpp`
- [PostgreSQL catalog](https://github.com/duckdb/ducklake/blob/main/src/metadata_manager/postgres_metadata_manager.cpp): `src/metadata_manager/postgres_metadata_manager.cpp`
- [Quack catalog](https://github.com/duckdb/ducklake/blob/main/src/metadata_manager/quack_metadata_manager.cpp): `src/metadata_manager/quack_metadata_manager.cpp`
- [Snapshot history](https://github.com/duckdb/ducklake/blob/main/src/common/ducklake_snapshot.cpp): `src/common/ducklake_snapshot.cpp`
- [Schema metadata](https://github.com/duckdb/ducklake/blob/main/src/storage/ducklake_schema_entry.cpp): `src/storage/ducklake_schema_entry.cpp`

### Other components

- **SQL user**
- **Parquet files**
- **Catalog database**

## Connections

- SQL user → DuckLake extension: attaches
- DuckLake extension → DuckLake catalog: registers storage (evidence: `src/ducklake_extension.cpp`)
- DuckLake extension → SQL functions: registers functions (evidence: `src/ducklake_extension.cpp`)
- SQL functions → DuckLake catalog: accesses catalog
- DuckLake catalog → Transactions: uses transactions
- Transactions → Metadata manager: records changes
- DuckLake catalog → Metadata manager: loads metadata
- Metadata manager → SQLite catalog: dispatches to (evidence: `src/storage/ducklake_metadata_manager.cpp`)
- Metadata manager → PostgreSQL catalog: dispatches to (evidence: `src/storage/ducklake_metadata_manager.cpp`)
- Metadata manager → Quack catalog: dispatches to (evidence: `src/storage/ducklake_metadata_manager.cpp`)
- Metadata manager → Catalog database: reads and writes
- DuckLake catalog → Table entries: manages entries
- Transactions → Snapshot history: tracks snapshots
- Metadata manager → Schema metadata: persists schemas (evidence: `src/storage/ducklake_metadata_manager.cpp`)
- SQL user → Change feed: queries changes
- Change feed → Snapshot history: reads history
- SQL user → Insert and writes: writes tables
- Insert and writes → Parquet files: writes files
- Insert and writes → Transactions: records files
- Add data files → Transactions: updates changes (evidence: `src/functions/ducklake_add_data_files.cpp`)
- SQL user → Table scanning: queries tables
- Table scanning → Multi-file reader: reads files
- Multi-file reader → Parquet files: reads data
- Table scanning → Table entries: scans table
- Table entries → Partitioning: uses partition spec
- Table entries → File statistics: provides statistics
- SQL user → Compaction: requests maintenance
- Compaction → Multi-file reader: reads files (evidence: `src/functions/ducklake_compaction_functions.cpp`)
- Compaction → Insert and writes: rewrites files (evidence: `src/functions/ducklake_compaction_functions.cpp`)
