How apache/hadoop works
This tree is a cross-section of Apache Hadoop: distributed storage, resource scheduling, RPC, and native client support. A common user workflow is accessing cloud object storage through Hadoop filesystem connectors; other branches cover NameNode maintenance, storage-policy satisfaction, and YARN queue and placement management. The RPC and libhdfs nodes show internal transport and native-client layers. The README provides no capability overview, so the map follows the tree and sampled source; several connector and service internals are represented at their enclosing directories where source was not sampled.
Main entry points: useQueueCardHandlers.ts Queue card handlers · namenode HDFS NameNode · thread_local_storage.h libhdfs client · GoogleHadoopFileSystem.java GCS filesystem
Components 24
Cloud storage
- GCS filesystem
hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleHadoopFileSystem.java - GCS storage adapter
hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java - Google Cloud Storage
- Baidu BOS connector
hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos - Tencent COS connector
hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn - Huawei OBS connector
hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs - Volcengine TOS connector
hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs
HDFS services
- HDFS NameNode
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode - Checkpoint retention
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorageRetentionManager.java - Policy manager
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps/StoragePolicySatisfyManager.java - Policy satisfier
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/sps
RPC transport
- Protobuf RPC engine
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java - RPC request header
hadoop-common-project/hadoop-common/src/main/proto/ProtobufRpcEngine.proto - RPC server dispatch
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java
Native client
- libhdfs client
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/thread_local_storage.h - Thread-local JNI state
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/thread_local_storage.h - Cyrus SASL engine
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/cyrus_sasl_engine.cc
YARN scheduling
- Queue card handlers
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueCardHandlers.ts - Scheduler UI state
- Queue actions
- Capacity editor
- Constraint manager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/PlacementConstraintManagerService.java
Other components
- Hadoop filesystem user
- Queue administrator
Connections 19
- Hadoop filesystem user → GCS filesystem: uses filesystem
- GCS filesystem → GCS storage adapter: accesses storage
- GCS storage adapter → Google Cloud Storage: reads and writes
- Hadoop filesystem user → Baidu BOS connector: uses filesystem
- Hadoop filesystem user → Tencent COS connector: uses filesystem
- Hadoop filesystem user → Huawei OBS connector: uses filesystem
- Hadoop filesystem user → Volcengine TOS connector: uses filesystem
- HDFS NameNode → Checkpoint retention: purges old state
- HDFS NameNode → Policy manager: manages policies
- Policy manager → Policy satisfier: instantiates service
- Protobuf RPC engine → RPC request header: marshals request
- RPC server dispatch → RPC request header: parses request
- libhdfs client → Thread-local JNI state: stores thread state
- Queue administrator → Queue card handlers: invokes actions
- Queue card handlers → Scheduler UI state: updates state
- Queue card handlers → Queue actions: requests updates
- Queue card handlers → Capacity editor: opens editor
- Queue administrator → Constraint manager: submits constraints
- libhdfs client → Cyrus SASL engine: authenticates RPC