# varunvilva-kickdrum/hadoop-hive-spark-hudi-airflow-jupyter-docker architecture

> Architecture diagram of the GitHub repository varunvilva-kickdrum/hadoop-hive-spark-hudi-airflow-jupyter-docker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/varunvilva-kickdrum/hadoop-hive-spark-hudi-airflow-jupyter-docker
- Repository: https://github.com/varunvilva-kickdrum/hadoop-hive-spark-hudi-airflow-jupyter-docker

## Overview

##### Project type and purpose
- This is a container-based big-data processing environment orchestrated via Docker Compose.
- Purpose: Stand up an end-to-end data platform with Hadoop (HDFS), Spark, Hive (with Postgres metastore), Jupyter Notebook, and Airflow for workflow automation.

##### File structure
- Top-level services defined in docker-compose.yml and sub-directories:
- hadoop/ (namenode + datanodes, HDFS init scripts)
- spark/ (Spark master & workers Dockerfile)
- hive/ (HiveServer2, hive-site.xml, Postgres metastore)
- jupyter/ (Notebook server with PySpark kernel)
- airflow/ (Airflow webserver, scheduler, worker, triggerer, flower, DAGs)
- Scripts to start subsets of the stack: start-essential.sh, start-spark.sh, start-airflow.sh
- Configuration/env files for container start order and resource sizing.

##### README for insights
- Container startup order: Namenode → Datanodes → Spark master → Spark workers → Postgres → Hive → Jupyter → Airflow
- Service UIs and ports listed
- Workflow: Load data into HDFS, process via Spark (from Jupyter or Airflow), query via Hive.

##### Main components & relationships
a. Components
- HDFS Cluster: Namenode + 2 Datanodes
- Spark Cluster: Master + 2 Workers
- HiveServer2 + PostgreSQL Metastore
- Jupyter Notebook Server (PySpark kernel)
- Airflow: Webserver, Scheduler, Worker, Triggerer, Flower
b. Interactions
- All compute layers (Spark, Hive, Jupyter) read/write data on HDFS
- HiveServer2 uses Postgres for metadata storage
- Jupyter submits Spark jobs to Spark master (spark://spark-master:7077)
- Airflow schedules and triggers SparkSubmitOperators, connecting to Spark master and HDFS, and can also interact with Hive
c. Architectural patterns
- Distributed data storage (HDFS) feeding multiple compute engines
- Containerized microservices for each component
- Decoupled workflow orchestration via Airflow
- Shared network layer in Docker Compose for service discovery
d. Key technologies
- Hadoop HDFS, Spark, Hive, PostgreSQL, Jupyter (with PySpark), Airflow

- Annotations for service dependencies (e.g., “depends_on” in docker-compose)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer (HDFS Cluster)"
        direction TB
        HDFSCluster["HDFS Cluster\n- NameNode\n- 2× DataNodes"]:::storage
    end

    %% Metadata Layer
    subgraph "Metadata Layer"
        direction TB
        Postgres["PostgreSQL Metastore"]:::metadata
        HiveServer2["HiveServer2"]:::compute
    end

    %% Compute Layer
    subgraph "Compute Layer"
        direction TB
        SparkMaster["Spark Master"]:::compute
        SparkWorkers["2× Spark Workers"]:::compute
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer (Airflow)"
        direction TB
        AirflowWeb["Webserver"]:::orchestration
        AirflowSched["Scheduler"]:::orchestration
        AirflowWorker["Worker"]:::orchestration
        AirflowTrig["Triggerer"]:::orchestration
        AirflowFlower["Flower"]:::orchestration
    end

    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        Jupyter["Jupyter Notebook"]:::client
    end

    %% Service Composition
    Compose["Docker Compose"]:::external

    %% Connections
    SparkMaster <-->|read/write| HDFSCluster
    HiveServer2 <-->|external tables| HDFSCluster
    HiveServer2 <-->|metadata| Postgres
    Jupyter -->|submit jobs| SparkMaster
    AirflowSched -->|spark_submit| SparkMaster
    AirflowWorker -->|tasks to HDFS/Hive| HDFSCluster
    AirflowWorker -->|tasks to HDFS/Hive| HiveServer2

    %% Click Events
    click HDFSCluster "https://github.com/varunvilva-kickdrum/hadoop-hive-spark-hudi-airflow-jupyter-docker/blob/main/hadoop/init-hdfs.sh"
    click SparkMaster "https://github.com/varunvilva-kickdrum/hadoop-hive-spark-hudi-airflow-jupyter-docker/tree/main/spark/Dockerfile"
    click HiveServer2 "https://github.com/varunvilva-kickdrum/hadoop-hive-spark-hudi-airflow-jupyter-docker/tree/main/hive/Dockerfile"
    click Postgres "https://github.com/varunvilva-kickdrum/hadoop-hive-spark-hudi-airflow-jupyter-docker/tree/main/postgres/Dockerfile"
    click Jupyter "https://github.com/varunvilva-kickdrum/hadoop-hive-spark-hudi-airflow-jupyter-docker/tree/main/jupyter/Dockerfile"
    click AirflowSched "https://github.com/varunvilva-kickdrum/hadoop-hive-spark-hudi-airflow-jupyter-docker/blob/main/airflow/Dockerfile.custom"
    click Compose "https://github.com/varunvilva-kickdrum/hadoop-hive-spark-hudi-airflow-jupyter-docker/blob/main/docker-compose.yml"

    %% Styles
    classDef storage fill:#BDD7EE,stroke:#5B9BD5,color:#1F4E79;
    classDef compute fill:#C6EFCE,stroke:#6AA84F,color:#274E13;
    classDef metadata fill:#FCE4D6,stroke:#E97451,color:#7F2F00;
    classDef orchestration fill:#EAD1DC,stroke:#A64D79,color:#4B1132;
    classDef client fill:#FFF2CC,stroke:#FFD966,color:#7F6000;
    classDef external fill:#D9D9D9,stroke:#A6A6A6,color:#000000;
```
