How linkcoder/hadoop-docker-cluster works
The project you’re looking at is essentially a Docker-based cluster setup for big data components. The repository is structured to build and run integrated services such as Hadoop, HBase, Hive, Spark, Zookeeper, MySQL (supporting Hive), and a basic CentOS with JDK8 image. The overall goal is to create a multi-node cluster through Docker and docker-compose, enabling a simulated “hadoop-docker-cluster” environment that replicates a real-world Hadoop ecosystem.
Main Components
• Docker Base Image (centos7+jdk8): This directory contains the Dockerfile and configuration files (bashrc, ssh_config) needed to set up a custom CentOS image with JDK8. It acts as a common base for several other images.
• Hadoop: Contains everything needed to deploy the Hadoop ecosystem – Dockerfile, configuration files (core-site.xml, hdfs-site.xml, etc.), and its own docker-compose file. Represent both NameNode and DataNodes here.
• HBase: Includes the configuration and Docker-compose file for setting up HBase which might be integrated with Hadoop for storage/processing.
• Hive: Provides Docker setup and configurations (alongside MySQL connectors) to run Hive on top of Hadoop/HBase/Hive Metastore using MySQL.
• MySQL: Acts as the backing database service for Hive Metastore (and possibly other services), with a significant data directory structure.
• Spark: Contains Dockerfile, configurations (log4j, spark-env.sh, etc.) along with a compose setup to deploy a Spark cluster.
• Zookeeper: Provides the coordination service for some of the components (commonly used with HBase, Hive, and Hadoop-related services). It includes Dockerfiles, configuration (zoo.cfg), and persistent data directories.
• Utility Scripts: “build.sh” and “in-container.sh” handle image building and container internal operations.
Understand the Relationships and Interactions
• The Docker images are built and managed via the provided shell scripts and docker-compose setups. Each component is a container and they communicate over a common network (e.g., “hadoop” network).
• Hadoop components (NameNode, DataNodes, ResourceManager, etc.) interact internally and expose web UIs (ports 50070 for HDFS and 8088 for Yarn).
• Zookeeper, critical for coordination, is clustered (multiple zoo directories for zk1, zk2, zk3) and signals state between services that rely on distributed consensus.
• HBase and Hive are layered on top of Hadoop; Hive uses MySQL for its metastore, while HBase leverages Hadoop’s HDFS as its file storage.
• Spark can serve as both a processing engine and a complementary service to Hadoop, typically interfacing with HDFS.
Architectural Patterns & Design Principles
• Containerization & Microservices: Each service is containerized and managed via docker-compose. This suggests loose coupling and service isolation.
• Configuration as Code: Each subdirectory comes with its own Dockerfile and dedicated configuration files, making the system modular and replicable.
• Separation of Concerns: The Docker images for distinct services ensure responsibilities (data storage for MySQL, processing for Spark, file system for HDFS, etc.) do not overlap.
Relevant Technologies and Frameworks
• Docker & Docker-compose for container orchestration.
• Hadoop ecosystem components for distributed storage and processing.
• Zookeeper for distributed coordination.
• Spark for big data analytics.
• MySQL for relational data storage used by Hive.
• Shell scripts (build.sh, in-container.sh) for automation tasks.
Putting It All Together
• Notes on external dependencies (e.g., instructions for setting up the host directories associated with volume mounts).