# akstrov/bigdatalabs architecture

> Architecture diagram of the GitHub repository akstrov/bigdatalabs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/akstrov/bigdatalabs
- Repository: https://github.com/akstrov/bigdatalabs

## Overview

##### Project type and purpose
- This is an educational “Big Data Labs” repository, not a production-grade system.
- Purpose: provide hands-on labs on distributed storage (HDFS/HBase), batch processing (MapReduce/Spark), and streaming (Kafka + Hadoop streaming).

##### File structure
- Top-level directories:
- lab0/ – Docker Compose environment setup for Hadoop, HBase, Spark, Kafka, Jupyter.
- lab2/ – A Maven-based Java project exercising basic HDFS operations (write, read, status).
- lab3_mapreduce/ – A Maven Java MapReduce WordCount example plus Python streaming mapper/reducer scripts.
- Configuration files under lab0/image/config → one container per service (config for core-site.xml, hdfs-site.xml, etc.).
- Build artifacts (lab2/target, lab3_mapreduce/target) indicate the compile-then-package workflow using Maven.

##### README for additional insights
- Confirms tech stack: Apache Hadoop (HDFS, YARN), HBase, Spark, Kafka, Docker, Maven, Java, Python.
- Describes lab progression:
- Lab 0: environment bootstrapping
- Lab 2: HDFS API exercises
- Lab 3: MapReduce (Java + Python streaming)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Developer Tools
    subgraph "Developer Tools"
        direction TB
        Maven["Maven & JDK"]:::dev
        Python3["Python3"]:::dev
    end

    %% Build & Artifacts
    subgraph "Build & Artifacts"
        direction TB
        lab2Build["lab2 Build (HadoopFileStatus.jar)"]:::artifact
        lab3Build["lab3 Build (WordCount.jar)"]:::artifact
        streamingScripts["Streaming Scripts (mapper.py, reducer.py)"]:::artifact
    end

    Maven -->|builds| lab2Build
    Maven -->|builds| lab3Build
    Python3 -->|runs| streamingScripts

    %% Environment Layer
    subgraph "Environment Layer"
        direction TB
        DockerCompose["Docker Compose"]:::orch
        subgraph "Hadoop Cluster"
            direction TB
            NameNode["HDFS NameNode"]:::storage
            DataNodes["HDFS DataNodes"]:::storage
            ResourceManager["YARN ResourceManager"]:::compute
            NodeManagers["YARN NodeManagers"]:::compute
        end
        subgraph "HBase"
            direction TB
            HBaseMaster["HBase Master"]:::storage
            RegionServers["HBase RegionServers"]:::storage
        end
        subgraph "Spark"
            direction TB
            SparkMaster["Spark Master"]:::compute
            SparkWorkers["Spark Workers"]:::compute
        end
        subgraph "Kafka Streaming"
            direction TB
            Zookeeper["Zookeeper"]:::streaming
            KafkaBroker["Kafka Broker"]:::streaming
        end
        Jupyter["Jupyter Notebook"]:::compute
    end

    %% Data Flow
    lab2Build -->|"submit JAR"| ResourceManager
    lab3Build -->|"submit JAR"| ResourceManager
    streamingScripts -->|"Hadoop Streaming"| NodeManagers
    Jupyter -->|"interactive jobs"| SparkMaster
    SparkMaster -->|"schedules tasks"| SparkWorkers
    SparkWorkers -->|"read/write"| DataNodes
    ResourceManager -->|"launches tasks on"| NodeManagers
    NodeManagers -->|"read/write"| DataNodes
    NameNode -->|"metadata RPC"| DataNodes
    KafkaBroker -->|"stream"| streamingScripts

    %% Orchestration
    DockerCompose --> NameNode
    DockerCompose --> DataNodes
    DockerCompose --> ResourceManager
    DockerCompose --> NodeManagers
    DockerCompose --> HBaseMaster
    DockerCompose --> RegionServers
    DockerCompose --> SparkMaster
    DockerCompose --> SparkWorkers
    DockerCompose --> Zookeeper
    DockerCompose --> KafkaBroker
    DockerCompose --> Jupyter

    %% Click Events
    click DockerCompose "https://github.com/akstrov/bigdatalabs/blob/main/lab0/docker-compose.yml"
    click Dockerfile "https://github.com/akstrov/bigdatalabs/tree/main/lab0/image/Dockerfile"
    click ConfigDir "https://github.com/akstrov/bigdatalabs/tree/main/lab0/image/config/"
    click CoreSite "https://github.com/akstrov/bigdatalabs/blob/main/lab0/image/config/core-site.xml"
    click HdfsSite "https://github.com/akstrov/bigdatalabs/blob/main/lab0/image/config/hdfs-site.xml"
    click StartHadoop "https://github.com/akstrov/bigdatalabs/blob/main/lab0/image/config/start-hadoop.sh"
    click YarnSite "https://github.com/akstrov/bigdatalabs/blob/main/lab0/image/config/yarn-site.xml"
    click MapredSite "https://github.com/akstrov/bigdatalabs/blob/main/lab0/image/config/mapred-site.xml"
    click HbaseSite "https://github.com/akstrov/bigdatalabs/blob/main/lab0/image/config/hbase-site.xml"
    click HbaseEnv "https://github.com/akstrov/bigdatalabs/blob/main/lab0/image/config/hbase-env.sh"
    click SparkDefaults "https://github.com/akstrov/bigdatalabs/blob/main/lab0/image/config/spark-defaults.conf"
    click StartSpark "https://github.com/akstrov/bigdatalabs/blob/main/lab0/image/config/start-spark.sh"
    click StartKafkaZk "https://github.com/akstrov/bigdatalabs/blob/main/lab0/image/config/start-kafka-zookeeper.sh"
    click StartJupyter "https://github.com/akstrov/bigdatalabs/blob/main/lab0/image/config/start-jupyter.sh"
    click Lab2Pom "https://github.com/akstrov/bigdatalabs/blob/main/lab2/hadoop_lab/pom.xml"
    click HDFSWrite "https://github.com/akstrov/bigdatalabs/blob/main/lab2/hadoop_lab/src/main/java/edu/ismagi/hadoop/hdfslab/HDFSWrite.java"
    click ReadHDFS "https://github.com/akstrov/bigdatalabs/blob/main/lab2/hadoop_lab/src/main/java/edu/ismagi/hadoop/hdfslab/ReadHDFS.java"
    click HadoopFileStatusSrc "https://github.com/akstrov/bigdatalabs/blob/main/lab2/hadoop_lab/src/main/java/edu/ismagi/hadoop/hdfslab/HadoopFileStatus.java"
    click Lab2Jar "https://github.com/akstrov/bigdatalabs/blob/main/lab2/hadoop_lab/target/HadoopFileStatus.jar"
    click Lab3Pom "https://github.com/akstrov/bigdatalabs/blob/main/lab3_mapreduce/pom.xml"
    click TokenizerMapper "https://github.com/akstrov/bigdatalabs/blob/main/lab3_mapreduce/src/main/java/edu/ismagi/hadoop/mapreduce/TokenizerMapper.java"
    click IntSumReducer "https://github.com/akstrov/bigdatalabs/blob/main/lab3_mapreduce/src/main/java/edu/ismagi/hadoop/mapreduce/IntSumReducer.java"
    click WordCountSrc "https://github.com/akstrov/bigdatalabs/blob/main/lab3_mapreduce/src/main/java/edu/ismagi/hadoop/mapreduce/WordCount.java"
    click MapperPy "https://github.com/akstrov/bigdatalabs/blob/main/lab3_mapreduce/mapper.py"
    click ReducerPy "https://github.com/akstrov/bigdatalabs/blob/main/lab3_mapreduce/reducer.py"

    %% Styles
    classDef dev fill:#EEEEEE,stroke:#333,stroke-width:1px
    classDef artifact fill:#F0F0F0,stroke:#666,stroke-width:1px
    classDef orchestration fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef storage fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef compute fill:#90EE90,stroke:#333,stroke-width:1px
    classDef streaming fill:#FFA500,stroke:#333,stroke-width:1px
```
