# linkcoder/hadoop-docker-cluster architecture

> Architecture diagram of the GitHub repository linkcoder/hadoop-docker-cluster, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/linkcoder/hadoop-docker-cluster
- Repository: https://github.com/linkcoder/hadoop-docker-cluster

## Overview

The project you’re looking at is essentially a Docker-based cluster setup for big data components. The repository is structured to build and run integrated services such as Hadoop, HBase, Hive, Spark, Zookeeper, MySQL (supporting Hive), and a basic CentOS with JDK8 image. The overall goal is to create a multi-node cluster through Docker and docker-compose, enabling a simulated “hadoop-docker-cluster” environment that replicates a real-world Hadoop ecosystem.

##### Main Components
- Docker Base Image (centos7+jdk8): This directory contains the Dockerfile and configuration files (bashrc, ssh_config) needed to set up a custom CentOS image with JDK8. It acts as a common base for several other images.
- Hadoop: Contains everything needed to deploy the Hadoop ecosystem – Dockerfile, configuration files (core-site.xml, hdfs-site.xml, etc.), and its own docker-compose file. Represent both NameNode and DataNodes here.
- HBase: Includes the configuration and Docker-compose file for setting up HBase which might be integrated with Hadoop for storage/processing.
- Hive: Provides Docker setup and configurations (alongside MySQL connectors) to run Hive on top of Hadoop/HBase/Hive Metastore using MySQL.
- MySQL: Acts as the backing database service for Hive Metastore (and possibly other services), with a significant data directory structure.
- Spark: Contains Dockerfile, configurations (log4j, spark-env.sh, etc.) along with a compose setup to deploy a Spark cluster.
- Zookeeper: Provides the coordination service for some of the components (commonly used with HBase, Hive, and Hadoop-related services). It includes Dockerfiles, configuration (zoo.cfg), and persistent data directories.
- Utility Scripts: “build.sh” and “in-container.sh” handle image building and container internal operations.

##### Understand the Relationships and Interactions
- The Docker images are built and managed via the provided shell scripts and docker-compose setups. Each component is a container and they communicate over a common network (e.g., “hadoop” network).
- Hadoop components (NameNode, DataNodes, ResourceManager, etc.) interact internally and expose web UIs (ports 50070 for HDFS and 8088 for Yarn).
- Zookeeper, critical for coordination, is clustered (multiple zoo directories for zk1, zk2, zk3) and signals state between services that rely on distributed consensus.
- HBase and Hive are layered on top of Hadoop; Hive uses MySQL for its metastore, while HBase leverages Hadoop’s HDFS as its file storage.
- Spark can serve as both a processing engine and a complementary service to Hadoop, typically interfacing with HDFS.

##### Architectural Patterns & Design Principles
- Containerization & Microservices: Each service is containerized and managed via docker-compose. This suggests loose coupling and service isolation.
- Configuration as Code: Each subdirectory comes with its own Dockerfile and dedicated configuration files, making the system modular and replicable.
- Separation of Concerns: The Docker images for distinct services ensure responsibilities (data storage for MySQL, processing for Spark, file system for HDFS, etc.) do not overlap.

##### Relevant Technologies and Frameworks
- Docker & Docker-compose for container orchestration.
- Hadoop ecosystem components for distributed storage and processing.
- Zookeeper for distributed coordination.
- Spark for big data analytics.
- MySQL for relational data storage used by Hive.
- Shell scripts (build.sh, in-container.sh) for automation tasks.

##### Putting It All Together
- Notes on external dependencies (e.g., instructions for setting up the host directories associated with volume mounts).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Client
    ExternalClient["External Client"]:::client

    %% Automation Scripts
    subgraph "Automation"
        Build["build.sh"]:::automation
        InContainer["in-container.sh"]:::automation
    end

    %% Docker Base Image
    Base["Docker Base Image (CentOS+jdk8)"]:::base

    %% Hadoop Cluster
    subgraph "Hadoop Cluster"
        NN["Hadoop NameNode (50070)"]:::processing
        DN["Hadoop DataNode"]:::processing
        RM["Hadoop ResourceManager (8088)"]:::processing
    end

    %% HBase Cluster
    subgraph "HBase Cluster"
        HB["HBase Service"]:::processing
    end

    %% Hive Service
    subgraph "Hive Service"
        HV["Hive Metastore & Service"]:::processing
    end

    %% MySQL Database
    subgraph "MySQL Database"
        SQL["MySQL DB"]:::storage
    end

    %% Spark Cluster
    subgraph "Spark Cluster"
        SM["Spark Master"]:::processing
        SW["Spark Worker"]:::processing
    end

    %% Zookeeper Cluster
    subgraph "Zookeeper Cluster"
        ZK1["Zookeeper - zk1"]:::coordination
        ZK2["Zookeeper - zk2"]:::coordination
        ZK3["Zookeeper - zk3"]:::coordination
    end

    %% Build process & Base Image
    Build -->|"builds"| Base

    %% Base Image used for all container images
    Base -->|"baseFor"| NN
    Base -->|"baseFor"| HB
    Base -->|"baseFor"| HV
    Base -->|"baseFor"| SM
    Base -->|"baseFor"| ZK1
    Base -->|"baseFor"| SQL

    %% Internal Interactions
    NN -->|"providesHDFS"| HB
    NN -->|"providesHDFS"| HV
    HV -->|"metastoreConn"| SQL
    SM -->|"readsW_HDFS"| NN
    SW -->|"readsW_HDFS"| NN
    SM -->|"integratesW_HBase"| HB

    %% Zookeeper Coordination (using one representative connection)
    ZK1 -->|"coordination"| NN
    ZK1 -->|"coordination"| HB
    ZK1 -->|"coordination"| HV
    ZK1 -->|"coordination"| RM

    %% External Client accessing Web UIs
    ExternalClient -->|"accesses"| NN
    ExternalClient -->|"accesses"| RM
    ExternalClient -->|"accesses"| HV

    %% Click Events
    click Base "https://github.com/linkcoder/hadoop-docker-cluster/tree/master/centos7+jdk8/"
    click NN "https://github.com/linkcoder/hadoop-docker-cluster/tree/master/hadoop/"
    click HB "https://github.com/linkcoder/hadoop-docker-cluster/tree/master/hbase/"
    click HV "https://github.com/linkcoder/hadoop-docker-cluster/tree/master/hive/"
    click SQL "https://github.com/linkcoder/hadoop-docker-cluster/tree/master/mysql/"
    click SM "https://github.com/linkcoder/hadoop-docker-cluster/tree/master/spark/"
    click ZK1 "https://github.com/linkcoder/hadoop-docker-cluster/tree/master/zookeeper/"
    click Build "https://github.com/linkcoder/hadoop-docker-cluster/blob/master/build.sh"
    click InContainer "https://github.com/linkcoder/hadoop-docker-cluster/blob/master/in-container.sh"

    %% Styles
    classDef base fill:#FFEEAD,stroke:#333,stroke-width:2px;
    classDef processing fill:#C5E1A5,stroke:#333,stroke-width:2px;
    classDef storage fill:#AED581,stroke:#333,stroke-width:2px;
    classDef coordination fill:#81D4FA,stroke:#333,stroke-width:2px;
    classDef automation fill:#CE93D8,stroke:#333,stroke-width:2px;
    classDef client fill:#FFE082,stroke:#333,stroke-width:2px;
```
