How apache/hudi works
Apache Hudi is an open table-format lakehouse platform. A typical flow brings batch, streaming, or database-change data through Spark/Flink ingestion, applies atomic writes and indexing, then records table history and data in cloud storage for query engines. Table services maintain layout and retention; catalog synchronization exposes table metadata to external catalogs. The graph combines README-documented capabilities with source paths from the tree; unsampled modules are represented at subsystem level, so only documented relationships are shown.
Main entry points: BaseHoodieWriteClient.java Write client · HoodieTableMetaClient.java Timeline metadata · HoodieStorage.java Storage abstraction · async Table services
Components 15
Ingestion and writes
- Ingestion tools
hudi-utilities/src/main/java/org/apache/hudi/utilities/ingestion/HoodieIngestionService.java - Write client
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java - Key generation
hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java - Indexing
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index
Table state and storage
- Timeline metadata
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java - Storage abstraction
hudi-io/src/main/java/org/apache/hudi/storage/HoodieStorage.java - Table services
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async
Queries and management
- Table queries
hudi-spark-datasource/hudi-spark-common/src/main/java - Trino connector
hudi-trino/src/main/java/io/trino/plugin/hudi/HudiSplitManager.java - Catalog sync
hudi-sync
Engine and cloud integrations
- Spark engine
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java - Flink engine
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java - Cloud catalogs
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java
Other components
- Data sources
- Cloud storage
Connections 16
- Data sources → Ingestion tools: supplies data
README.md - Ingestion tools → Write client: ingests through
README.md - Write client → Key generation: generates keys
README.md - Write client → Indexing: maintains index
README.md - Write client → Timeline metadata: commits atomically
README.md - Write client → Storage abstraction: writes table data
README.md - Storage abstraction → Cloud storage: stores data
README.md - Table services → Storage abstraction: optimizes layout
README.md - Table services → Timeline metadata: manages history
README.md - Table queries → Timeline metadata: reads table state
README.md - Table queries → Indexing: uses indexes
README.md - Trino connector → Timeline metadata: queries tables
- Catalog sync → Cloud catalogs: syncs metadata
README.md - Catalog sync → Timeline metadata: publishes table metadata
README.md - Spark engine → Write client: runs writes
README.md - Flink engine → Write client: runs writes
README.md