How duckdb/ducklake works
DuckLake is a DuckDB extension that presents a SQL-accessible lakehouse: catalog metadata lives in a database, while table data lives in Parquet files. The documented workflow is attach a catalog, create or modify tables, then query them; snapshots enable time travel and change-feed queries. The graph shows the principal storage, transaction, and metadata layers plus documented maintenance and metadata-query capabilities. Several implementation relationships are inferred only from the SOURCE INDEX reference lists, so unsampled subsystems and uncertain wiring are omitted.
Main entry points: ducklake_metadata_manager.cpp Metadata manager · ducklake_catalog.cpp DuckLake catalog · ducklake_transaction.cpp Transactions · ducklake_table_entry.cpp Table entries
Components 22
SQL integration
- DuckLake extension
src/ducklake_extension.cpp - SQL functions
src/functions - Change feed
src/functions/ducklake_table_changes.cpp
Catalog and transactions
- DuckLake catalog
src/storage/ducklake_catalog.cpp - Transactions
src/storage/ducklake_transaction.cpp - Table entries
src/storage/ducklake_table_entry.cpp
Data operations
- Insert and writes
src/storage/ducklake_insert.cpp - Add data files
src/functions/ducklake_add_data_files.cpp - Table scanning
src/storage/ducklake_scan.cpp - Multi-file reader
src/storage/ducklake_multi_file_reader.cpp - Partitioning
src/storage/ducklake_partition_data.cpp - File statistics
src/storage/ducklake_stats.cpp - Compaction
src/functions/ducklake_compaction_functions.cpp
Metadata and history
- Metadata manager
src/storage/ducklake_metadata_manager.cpp - SQLite catalog
src/metadata_manager/sqlite_metadata_manager.cpp - PostgreSQL catalog
src/metadata_manager/postgres_metadata_manager.cpp - Quack catalog
src/metadata_manager/quack_metadata_manager.cpp - Snapshot history
src/common/ducklake_snapshot.cpp - Schema metadata
src/storage/ducklake_schema_entry.cpp
Other components
- SQL user
- Parquet files
- Catalog database
Connections 29
- SQL user → DuckLake extension: attaches
- DuckLake extension → DuckLake catalog: registers storage
src/ducklake_extension.cpp - DuckLake extension → SQL functions: registers functions
src/ducklake_extension.cpp - SQL functions → DuckLake catalog: accesses catalog
- DuckLake catalog → Transactions: uses transactions
- Transactions → Metadata manager: records changes
- DuckLake catalog → Metadata manager: loads metadata
- Metadata manager → SQLite catalog: dispatches to
src/storage/ducklake_metadata_manager.cpp - Metadata manager → PostgreSQL catalog: dispatches to
src/storage/ducklake_metadata_manager.cpp - Metadata manager → Quack catalog: dispatches to
src/storage/ducklake_metadata_manager.cpp - Metadata manager → Catalog database: reads and writes
- DuckLake catalog → Table entries: manages entries
- Transactions → Snapshot history: tracks snapshots
- Metadata manager → Schema metadata: persists schemas
src/storage/ducklake_metadata_manager.cpp - SQL user → Change feed: queries changes
- Change feed → Snapshot history: reads history
- SQL user → Insert and writes: writes tables
- Insert and writes → Parquet files: writes files
- Insert and writes → Transactions: records files
- Add data files → Transactions: updates changes
src/functions/ducklake_add_data_files.cpp - SQL user → Table scanning: queries tables
- Table scanning → Multi-file reader: reads files
- Multi-file reader → Parquet files: reads data
- Table scanning → Table entries: scans table
- Table entries → Partitioning: uses partition spec
- Table entries → File statistics: provides statistics
- SQL user → Compaction: requests maintenance
- Compaction → Multi-file reader: reads files
src/functions/ducklake_compaction_functions.cpp - Compaction → Insert and writes: rewrites files
src/functions/ducklake_compaction_functions.cpp