# apache/hadoop architecture

> Architecture diagram of the GitHub repository apache/hadoop, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/hadoop
- Repository: https://github.com/apache/hadoop

## Overview

This tree is a cross-section of Apache Hadoop: distributed storage, resource scheduling, RPC, and native client support. A common user workflow is accessing cloud object storage through Hadoop filesystem connectors; other branches cover NameNode maintenance, storage-policy satisfaction, and YARN queue and placement management. The RPC and libhdfs nodes show internal transport and native-client layers. The README provides no capability overview, so the map follows the tree and sampled source; several connector and service internals are represented at their enclosing directories where source was not sampled.

Main entry points: [`hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueCardHandlers.ts`](https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueCardHandlers.ts) (Queue card handlers), [`hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode`](https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode) (HDFS NameNode), [`hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/thread_local_storage.h`](https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/thread_local_storage.h) (libhdfs client), [`hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleHadoopFileSystem.java`](https://github.com/apache/hadoop/blob/trunk/hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleHadoopFileSystem.java) (GCS filesystem)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_cloud["Cloud storage"]
  node_gcsfs["GCS filesystem"]
  node_gcsadapter["GCS storage adapter"]
  node_gcsservice[("Google Cloud Storage")]
  node_bos["Baidu BOS connector"]
  node_cos["Tencent COS connector"]
  node_obs["Huawei OBS connector"]
  node_tos["Volcengine TOS connector"]
end

subgraph group_hdfs["HDFS services"]
  node_namenode["HDFS NameNode"]
  node_retention["Checkpoint retention"]
  node_spsmanager["Policy manager"]
  node_spsservice["Policy satisfier"]
end

subgraph group_rpc["RPC transport"]
  node_rpcclient["Protobuf RPC engine"]
  node_rpcheader["RPC request header"]
  node_rpcserver["RPC server dispatch"]
end

subgraph group_native["Native client"]
  node_libhdfs["libhdfs client"]
  node_threadstate["Thread-local JNI state"]
  node_sasl["Cyrus SASL engine"]
end

subgraph group_yarn["YARN scheduling"]
  node_queuehandlers["Queue card handlers"]
  node_schedulerstore["Scheduler UI state"]
  node_queueactions["Queue actions"]
  node_capacityeditor["Capacity editor"]
  node_placement["Constraint manager"]
end

node_fsuser(("Hadoop filesystem user"))
node_queueuser(("Queue administrator"))

node_fsuser -->|"uses filesystem"| node_gcsfs
node_gcsfs -->|"accesses storage"| node_gcsadapter
node_gcsadapter -->|"reads and writes"| node_gcsservice
node_fsuser -->|"uses filesystem"| node_bos
node_fsuser -->|"uses filesystem"| node_cos
node_fsuser -->|"uses filesystem"| node_obs
node_fsuser -->|"uses filesystem"| node_tos
node_namenode -->|"purges old state"| node_retention
node_namenode -->|"manages policies"| node_spsmanager
node_spsmanager -->|"instantiates service"| node_spsservice
node_rpcclient -->|"marshals request"| node_rpcheader
node_rpcserver -->|"parses request"| node_rpcheader
node_libhdfs -->|"stores thread state"| node_threadstate
node_queueuser -->|"invokes actions"| node_queuehandlers
node_queuehandlers -->|"updates state"| node_schedulerstore
node_queuehandlers -->|"requests updates"| node_queueactions
node_queuehandlers -->|"opens editor"| node_capacityeditor
node_queueuser -.->|"submits constraints"| node_placement
node_libhdfs -.->|"authenticates RPC"| node_sasl

click node_gcsfs "https://github.com/apache/hadoop/blob/trunk/hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleHadoopFileSystem.java"
click node_gcsadapter "https://github.com/apache/hadoop/blob/trunk/hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java"
click node_bos "https://github.com/apache/hadoop/tree/trunk/hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos"
click node_cos "https://github.com/apache/hadoop/tree/trunk/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn"
click node_obs "https://github.com/apache/hadoop/tree/trunk/hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs"
click node_tos "https://github.com/apache/hadoop/tree/trunk/hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs"
click node_namenode "https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode"
click node_retention "https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorageRetentionManager.java"
click node_spsmanager "https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/StoragePolicySatisfyManager.java"
click node_spsservice "https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps"
click node_rpcclient "https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java"
click node_rpcheader "https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/proto/ProtobufRpcEngine.proto"
click node_rpcserver "https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java"
click node_libhdfs "https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/thread_local_storage.h"
click node_threadstate "https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/thread_local_storage.h"
click node_sasl "https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/cyrus_sasl_engine.cc"
click node_queuehandlers "https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueCardHandlers.ts"
click node_placement "https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/PlacementConstraintManagerService.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_gcsfs,node_gcsadapter,node_gcsservice,node_bos,node_cos,node_obs,node_tos,node_fsuser toneBlue
class node_namenode,node_retention,node_spsmanager,node_spsservice toneAmber
class node_rpcclient,node_rpcheader,node_rpcserver toneMint
class node_libhdfs,node_threadstate,node_sasl,node_queueuser toneRose
class node_queuehandlers,node_schedulerstore,node_queueactions,node_capacityeditor,node_placement toneIndigo
```

## Components

### Cloud storage

- [GCS filesystem](https://github.com/apache/hadoop/blob/trunk/hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleHadoopFileSystem.java): `hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleHadoopFileSystem.java`
- [GCS storage adapter](https://github.com/apache/hadoop/blob/trunk/hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java): `hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java`
- **Google Cloud Storage**
- [Baidu BOS connector](https://github.com/apache/hadoop/tree/trunk/hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos): `hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos`
- [Tencent COS connector](https://github.com/apache/hadoop/tree/trunk/hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn): `hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn`
- [Huawei OBS connector](https://github.com/apache/hadoop/tree/trunk/hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs): `hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs`
- [Volcengine TOS connector](https://github.com/apache/hadoop/tree/trunk/hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs): `hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs`

### HDFS services

- [HDFS NameNode](https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode): `hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode`
- [Checkpoint retention](https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorageRetentionManager.java): `hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorageRetentionManager.java`
- [Policy manager](https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/StoragePolicySatisfyManager.java): `hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/StoragePolicySatisfyManager.java`
- [Policy satisfier](https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps): `hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps`

### RPC transport

- [Protobuf RPC engine](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java): `hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java`
- [RPC request header](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/proto/ProtobufRpcEngine.proto): `hadoop-common-project/hadoop-common/src/main/proto/ProtobufRpcEngine.proto`
- [RPC server dispatch](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java): `hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java`

### Native client

- [libhdfs client](https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/thread_local_storage.h): `hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/thread_local_storage.h`
- [Thread-local JNI state](https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/thread_local_storage.h): `hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/thread_local_storage.h`
- [Cyrus SASL engine](https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/cyrus_sasl_engine.cc): `hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/cyrus_sasl_engine.cc`

### YARN scheduling

- [Queue card handlers](https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueCardHandlers.ts): `hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueCardHandlers.ts`
- **Scheduler UI state**
- **Queue actions**
- **Capacity editor**
- [Constraint manager](https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/PlacementConstraintManagerService.java): `hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/PlacementConstraintManagerService.java`

### Other components

- **Hadoop filesystem user**
- **Queue administrator**

## Connections

- Hadoop filesystem user → GCS filesystem: uses filesystem
- GCS filesystem → GCS storage adapter: accesses storage
- GCS storage adapter → Google Cloud Storage: reads and writes
- Hadoop filesystem user → Baidu BOS connector: uses filesystem
- Hadoop filesystem user → Tencent COS connector: uses filesystem
- Hadoop filesystem user → Huawei OBS connector: uses filesystem
- Hadoop filesystem user → Volcengine TOS connector: uses filesystem
- HDFS NameNode → Checkpoint retention: purges old state
- HDFS NameNode → Policy manager: manages policies
- Policy manager → Policy satisfier: instantiates service
- Protobuf RPC engine → RPC request header: marshals request
- RPC server dispatch → RPC request header: parses request
- libhdfs client → Thread-local JNI state: stores thread state
- Queue administrator → Queue card handlers: invokes actions
- Queue card handlers → Scheduler UI state: updates state
- Queue card handlers → Queue actions: requests updates
- Queue card handlers → Capacity editor: opens editor
- Queue administrator → Constraint manager: submits constraints
- libhdfs client → Cyrus SASL engine: authenticates RPC