# apache/hudi architecture

> Architecture diagram of the GitHub repository apache/hudi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/hudi
- Repository: https://github.com/apache/hudi

## Overview

Apache Hudi is an open table-format lakehouse platform. A typical flow brings batch, streaming, or database-change data through Spark/Flink ingestion, applies atomic writes and indexing, then records table history and data in cloud storage for query engines. Table services maintain layout and retention; catalog synchronization exposes table metadata to external catalogs. The graph combines README-documented capabilities with source paths from the tree; unsampled modules are represented at subsystem level, so only documented relationships are shown.

Main entry points: [`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java`](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java) (Write client), [`hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java`](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java) (Timeline metadata), [`hudi-io/src/main/java/org/apache/hudi/storage/HoodieStorage.java`](https://github.com/apache/hudi/blob/master/hudi-io/src/main/java/org/apache/hudi/storage/HoodieStorage.java) (Storage abstraction), [`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async`](https://github.com/apache/hudi/tree/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async) (Table services)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion and writes"]
  node_ingestion["Ingestion tools"]
  node_write_client["Write client"]
  node_keygen["Key generation<br/>[KeyGenerator.java]"]
  node_index["Indexing"]
end

subgraph group_table["Table state and storage"]
  node_timeline["Timeline metadata"]
  node_storage_api["Storage abstraction<br/>[HoodieStorage.java]"]
  node_table_services["Table services"]
end

subgraph group_access["Queries and management"]
  node_query["Table queries"]
  node_trino["Trino connector"]
  node_catalog_sync["Catalog sync"]
end

subgraph group_integrations["Engine and cloud integrations"]
  node_spark_engine["Spark engine"]
  node_flink_engine["Flink engine"]
  node_cloud_catalogs["Cloud catalogs"]
end

node_data_sources(("Data sources"))
node_cloud_storage[("Cloud storage")]

node_data_sources -->|"supplies data"| node_ingestion
node_ingestion -->|"ingests through"| node_write_client
node_write_client -->|"generates keys"| node_keygen
node_write_client -->|"maintains index"| node_index
node_write_client -->|"commits atomically"| node_timeline
node_write_client -->|"writes table data"| node_storage_api
node_storage_api -->|"stores data"| node_cloud_storage
node_table_services -->|"optimizes layout"| node_storage_api
node_table_services -->|"manages history"| node_timeline
node_query -->|"reads table state"| node_timeline
node_query -->|"uses indexes"| node_index
node_trino -.->|"queries tables"| node_timeline
node_catalog_sync -->|"syncs metadata"| node_cloud_catalogs
node_catalog_sync -->|"publishes table metadata"| node_timeline
node_spark_engine -->|"runs writes"| node_write_client
node_flink_engine -->|"runs writes"| node_write_client

click node_ingestion "https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/ingestion/HoodieIngestionService.java"
click node_write_client "https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java"
click node_keygen "https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java"
click node_index "https://github.com/apache/hudi/tree/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index"
click node_timeline "https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java"
click node_storage_api "https://github.com/apache/hudi/blob/master/hudi-io/src/main/java/org/apache/hudi/storage/HoodieStorage.java"
click node_table_services "https://github.com/apache/hudi/tree/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async"
click node_query "https://github.com/apache/hudi/tree/master/hudi-spark-datasource/hudi-spark-common/src/main/java"
click node_trino "https://github.com/apache/hudi/blob/master/hudi-trino/src/main/java/io/trino/plugin/hudi/HudiSplitManager.java"
click node_catalog_sync "https://github.com/apache/hudi/tree/master/hudi-sync"
click node_spark_engine "https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java"
click node_flink_engine "https://github.com/apache/hudi/blob/master/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java"
click node_cloud_catalogs "https://github.com/apache/hudi/blob/master/hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingestion,node_write_client,node_keygen,node_index toneBlue
class node_timeline,node_storage_api,node_table_services,node_cloud_storage toneAmber
class node_query,node_trino,node_catalog_sync toneMint
class node_spark_engine,node_flink_engine,node_cloud_catalogs toneRose
class node_data_sources toneIndigo
```

## Components

### Ingestion and writes

- [Ingestion tools](https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/ingestion/HoodieIngestionService.java): `hudi-utilities/src/main/java/org/apache/hudi/utilities/ingestion/HoodieIngestionService.java`
- [Write client](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java): `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java`
- [Key generation](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java): `hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java`
- [Indexing](https://github.com/apache/hudi/tree/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index): `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index`

### Table state and storage

- [Timeline metadata](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java): `hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java`
- [Storage abstraction](https://github.com/apache/hudi/blob/master/hudi-io/src/main/java/org/apache/hudi/storage/HoodieStorage.java): `hudi-io/src/main/java/org/apache/hudi/storage/HoodieStorage.java`
- [Table services](https://github.com/apache/hudi/tree/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async): `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async`

### Queries and management

- [Table queries](https://github.com/apache/hudi/tree/master/hudi-spark-datasource/hudi-spark-common/src/main/java): `hudi-spark-datasource/hudi-spark-common/src/main/java`
- [Trino connector](https://github.com/apache/hudi/blob/master/hudi-trino/src/main/java/io/trino/plugin/hudi/HudiSplitManager.java): `hudi-trino/src/main/java/io/trino/plugin/hudi/HudiSplitManager.java`
- [Catalog sync](https://github.com/apache/hudi/tree/master/hudi-sync): `hudi-sync`

### Engine and cloud integrations

- [Spark engine](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java): `hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java`
- [Flink engine](https://github.com/apache/hudi/blob/master/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java): `hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java`
- [Cloud catalogs](https://github.com/apache/hudi/blob/master/hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java): `hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java`

### Other components

- **Data sources**
- **Cloud storage**

## Connections

- Data sources → Ingestion tools: supplies data (evidence: `README.md`)
- Ingestion tools → Write client: ingests through (evidence: `README.md`)
- Write client → Key generation: generates keys (evidence: `README.md`)
- Write client → Indexing: maintains index (evidence: `README.md`)
- Write client → Timeline metadata: commits atomically (evidence: `README.md`)
- Write client → Storage abstraction: writes table data (evidence: `README.md`)
- Storage abstraction → Cloud storage: stores data (evidence: `README.md`)
- Table services → Storage abstraction: optimizes layout (evidence: `README.md`)
- Table services → Timeline metadata: manages history (evidence: `README.md`)
- Table queries → Timeline metadata: reads table state (evidence: `README.md`)
- Table queries → Indexing: uses indexes (evidence: `README.md`)
- Trino connector → Timeline metadata: queries tables
- Catalog sync → Cloud catalogs: syncs metadata (evidence: `README.md`)
- Catalog sync → Timeline metadata: publishes table metadata (evidence: `README.md`)
- Spark engine → Write client: runs writes (evidence: `README.md`)
- Flink engine → Write client: runs writes (evidence: `README.md`)
