How apache/iceberg-rust works
Apache Iceberg Rust is a Rust implementation of the Iceberg table format, with catalog, storage, scan/read, write/transaction, and encryption capabilities. A typical workflow resolves a table through a catalog, accesses its metadata and data files through storage, and reads or writes table data while managing snapshots. The tree and README establish the product boundaries; sampled excerpts confirm catalog interfaces, scan context, Arrow reading, REST authentication, encryption, and storage behavior. Wiring between unsampled subsystems is shown only where supported by those boundaries.
Main entry points: session.rs Catalog interface · catalog.rs REST catalog · client.rs REST client · table_metadata.rs Table metadata
Components 25
Catalogs
- Catalog interface
crates/iceberg/src/catalog/session.rs - Catalog loader
crates/catalog/loader/src/lib.rs - REST catalog
crates/catalog/rest/src/catalog.rs - REST client
crates/catalog/rest/src/client.rs - OAuth2 auth
crates/catalog/rest/src/auth/oauth2.rs - Glue catalog
crates/catalog/glue/src/catalog.rs - Hive catalog
crates/catalog/hms/src/catalog.rs - S3 Tables catalog
crates/catalog/s3tables/src/catalog.rs - SQL catalog
crates/catalog/sql/src/catalog.rs
Table Model
- Table metadata
crates/iceberg/src/spec/table_metadata.rs - Schema model
crates/iceberg/src/spec/schema/mod.rs - Transactions and snapshots
crates/iceberg/src/transaction/mod.rs - Table inspection
crates/iceberg/src/inspect/mod.rs
Data Access
- Scan planning
crates/iceberg/src/scan/context.rs - Arrow reader
crates/iceberg/src/arrow/reader/pipeline.rs - Data and delete writers
crates/iceberg/src/writer/mod.rs - File I/O
crates/iceberg/src/io/file_io.rs - OpenDAL storage
crates/storage/opendal/src/lib.rs
Security
- Encryption manager
crates/iceberg/src/encryption/manager.rs - Key management interface
crates/iceberg/src/encryption/kms/client.rs
Bindings
- Python bindings
bindings/python/src/lib.rs
Other components
- Rust or Python consumer
- Catalog service
- Object or file storage
- External KMS
Connections 28
- Rust or Python consumer → Catalog loader: selects catalog
- Rust or Python consumer → Catalog interface: uses catalog
- Catalog loader → Catalog interface: loads implementation
- Catalog interface → REST catalog: dispatches REST
- Catalog interface → Glue catalog: dispatches Glue
- Catalog interface → Hive catalog: dispatches HMS
- Catalog interface → S3 Tables catalog: dispatches S3 Tables
- Catalog interface → SQL catalog: dispatches SQL
- REST catalog → REST client: sends requests
- REST client → OAuth2 auth: authenticates requests
- REST client → Catalog service: HTTP requests
- REST catalog → Table metadata: returns table
- Catalog interface → Table metadata: returns table
- Table metadata → Schema model: defines fields
- Rust or Python consumer → Scan planning: requests scan
- Scan planning → Arrow reader: provides file tasks
- Arrow reader → File I/O: reads data files
- Rust or Python consumer → Data and delete writers: writes table data
- Data and delete writers → File I/O: writes data files
- File I/O → OpenDAL storage: uses storage adapter
- OpenDAL storage → Object or file storage: reads and writes
- Rust or Python consumer → Transactions and snapshots: commits changes
- Transactions and snapshots → Catalog interface: updates table
- Rust or Python consumer → Table inspection: inspects table
- Rust or Python consumer → Encryption manager: uses encryption
- Encryption manager → Key management interface: wraps or unwraps keys
- Key management interface → External KMS: key operations
- Python bindings → Rust or Python consumer: exposes API