How varunvilva-kickdrum/hadoop-hive-spark-hudi-airflow-jupyter-docker works
Project type and purpose
• This is a container-based big-data processing environment orchestrated via Docker Compose.
• Purpose: Stand up an end-to-end data platform with Hadoop (HDFS), Spark, Hive (with Postgres metastore), Jupyter Notebook, and Airflow for workflow automation.
File structure
• Top-level services defined in docker-compose.yml and sub-directories:
• hadoop/ (namenode + datanodes, HDFS init scripts)
• spark/ (Spark master & workers Dockerfile)
• hive/ (HiveServer2, hive-site.xml, Postgres metastore)
• jupyter/ (Notebook server with PySpark kernel)
• airflow/ (Airflow webserver, scheduler, worker, triggerer, flower, DAGs)
• Scripts to start subsets of the stack: start-essential.sh, start-spark.sh, start-airflow.sh
• Configuration/env files for container start order and resource sizing.
README for insights
• Container startup order: Namenode → Datanodes → Spark master → Spark workers → Postgres → Hive → Jupyter → Airflow
• Service UIs and ports listed
• Workflow: Load data into HDFS, process via Spark (from Jupyter or Airflow), query via Hive.
Main components & relationships
a. Components
• HDFS Cluster: Namenode + 2 Datanodes
• Spark Cluster: Master + 2 Workers
• HiveServer2 + PostgreSQL Metastore
• Jupyter Notebook Server (PySpark kernel)
• Airflow: Webserver, Scheduler, Worker, Triggerer, Flower
b. Interactions
• All compute layers (Spark, Hive, Jupyter) read/write data on HDFS
• HiveServer2 uses Postgres for metadata storage
• Jupyter submits Spark jobs to Spark master (spark://spark-master:7077)
• Airflow schedules and triggers SparkSubmitOperators, connecting to Spark master and HDFS, and can also interact with Hive
c. Architectural patterns
• Distributed data storage (HDFS) feeding multiple compute engines
• Containerized microservices for each component
• Decoupled workflow orchestration via Airflow
• Shared network layer in Docker Compose for service discovery
d. Key technologies
• Hadoop HDFS, Spark, Hive, PostgreSQL, Jupyter (with PySpark), Airflow
• Annotations for service dependencies (e.g., “depends_on” in docker-compose)