# beingmani/mbd-assignment architecture

> Architecture diagram of the GitHub repository beingmani/mbd-assignment, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/beingmani/mbd-assignment
- Repository: https://github.com/beingmani/mbd-assignment

## Overview

##### Project type and purpose
- This is an educational Hadoop/MapReduce project containing:
– Assignment 1: word-count, unique-word pipelines (single job, chain mapper, two jobs)
– Assignment 2: a friend-recommender MapReduce job and a Python/Notebook exercise
– A MkDocs-generated documentation site

##### Main components
a. Developer workstation(s)
– Cloudera VM
– MacOS + Docker
– Python environment (for Jupyter/Ex1)
b. Build/artifact creation
– Java source (src/) → compiled .class → JARs under out/artifacts or jars/
– Python scripts / Jupyter notebooks
c. Hadoop cluster (real or Docker)
– HDFS (input directories; output directories)
– YARN/MapReduce framework
– JobManager (StubDriver.jar, WordCount.jar, Custom*.jar)
d. MapReduce execution
– Mapper classes
– Reducer classes
– (Optional) ChainMapper or two sequential jobs
– MRUnit tests (unit test harness)
e. Data flow
– Raw text files in HDFS → Map tasks → Shuffle & Sort → Reduce tasks → output files
f. Documentation site
– MkDocs (mkdocs.yml) → static HTML/CSS/JS under docs/site

##### Relationships & interactions
- Developer → builds JARs & Python scripts
- Developer → submits jobs (via StubDriver on JVM or python runner) → YARN
- YARN → schedules Map tasks on cluster nodes → read input blocks from HDFS
- Mappers → emit key-value pairs → shuffle across network → sorted by key
- Reducers → aggregate per key → write output files back to HDFS
- MRUnit tests run locally against mapper/reducer logic (no cluster)
- MkDocs builds documentation site; user browses via static web server

##### Architectural patterns / design principles
– Batch ETL pipeline (map → reduce → output)
– Single-responsibility: separate Mapper & Reducer classes
– Job orchestration: single job vs. chaining vs. multi-job workflows
– Environment abstraction: same code runs on Cloudera VM or MacOS/Docker
– Test harness decouples logic from Hadoop cluster

##### Technologies & frameworks
– Java 8+, Hadoop MapReduce API
– MRUnit for unit testing
– Python 3 + Jupyter for Ex1 in Assign2
– MkDocs (YAML config, Markdown → HTML)

– Clear labels for each component (e.g., “StubDriver.java” → “CustomWordCount.jar”)
– Optional: annotate key intermediate files (part-r-00000)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Developer Environments
    subgraph "Developer Workstations"
        vm["Cloudera VM"]:::dev
        mac["MacOS + Docker"]:::dev
        python_env["Python Environment (Jupyter)"]:::dev
    end

    %% Build Artifacts
    subgraph "Build & Source"
        src1["Exercise3 Java Source"]:::dev
        jar1["WordCount.jar"]:::compute
        src2["Exercise4 Java Source"]:::dev
        stub_src["StubDriver & MRUnit Tests"]:::dev
        jar2["CustomUniqueWordCount & UniqueWordsTwoJobs JARs"]:::compute
        py_scripts["ex1.ipynb / ex1.py"]:::dev
        fr_src["FriendRecommender.java Source"]:::dev
        fr_tests["MRUnit Tests"]:::dev
    end

    %% Hadoop Cluster Components
    subgraph "Hadoop Cluster"
        jobmgr["JobManager (YARN)"]:::compute
        map_task["Map Tasks"]:::compute
        shuffle["Shuffle & Sort"]:::compute
        reduce_task["Reduce Tasks"]:::compute
        hdfs_in1["HDFS Input (Ex3)"]:::storage
        hdfs_out1["HDFS Output (Ex3)"]:::storage
        hdfs_in2["HDFS Input (Ex4)"]:::storage
        hdfs_out2["HDFS Output (Ex4)"]:::storage
        hdfs_in3["HDFS Input (Ex3, Assign2)"]:::storage
        hdfs_out3["HDFS Output (Ex3, Assign2)"]:::storage
    end

    %% Documentation Site
    subgraph "Documentation Site"
        mkdocs_cfg["mkdocs.yml"]:::docs
        docs_src["Markdown Docs"]:::docs
        site["Generated Site"]:::docs
    end

    %% Data & Control Flows
    vm --> src1
    vm --> src2
    vm --> stub_src
    mac --> src1
    mac --> src2
    mac --> stub_src
    vm --> jar1
    src1 --> jar1
    src2 --> jar2
    stub_src --> jar2
    python_env --> py_scripts
    vm --> fr_src
    fr_src --> jobmgr
    fr_tests --> jobmgr

    jar1 --> jobmgr
    jar2 --> jobmgr

    hdfs_in1 --> map_task
    hdfs_in2 --> map_task
    hdfs_in3 --> map_task

    jobmgr --> map_task
    map_task --> shuffle
    shuffle --> reduce_task
    reduce_task --> hdfs_out1
    reduce_task --> hdfs_out2
    reduce_task --> hdfs_out3

    py_scripts -->|"Local Execution"| python_env

    mkdocs_cfg --> docs_src
    docs_src --> site

    %% Click Events
    click vm "https://github.com/beingmani/mbd-assignment/tree/main/Assign1/Cloudera"
    click mac "https://github.com/beingmani/mbd-assignment/tree/main/Assign1/MacOS"
    click src1 "https://github.com/beingmani/mbd-assignment/tree/main/Assign1/Cloudera/Exercise3/src"
    click jar1 "https://github.com/beingmani/mbd-assignment/blob/main/Assign1/Cloudera/Exercise3/jars/WordCount.jar"
    click hdfs_in1 "https://github.com/beingmani/mbd-assignment/tree/main/Assign1/Cloudera/Exercise3/input"
    click hdfs_out1 "https://github.com/beingmani/mbd-assignment/tree/main/Assign1/Cloudera/Exercise3/output"
    click src2 "https://github.com/beingmani/mbd-assignment/tree/main/Assign1/Cloudera/Exercise4/src/FreqCounter"
    click stub_src "https://github.com/beingmani/mbd-assignment/blob/main/Assign1/Cloudera/Exercise4/src/StubDriver.java"
    click stub_src "https://github.com/beingmani/mbd-assignment/blob/main/Assign1/Cloudera/Exercise4/src/StubTest.java"
    click jar2 "https://github.com/beingmani/mbd-assignment/tree/main/Assign1/Cloudera/Exercise4/jars"
    click hdfs_in2 "https://github.com/beingmani/mbd-assignment/tree/main/Assign1/Cloudera/Exercise4/input"
    click hdfs_out2 "https://github.com/beingmani/mbd-assignment/tree/main/Assign1/Cloudera/Exercise4/output"
    click py_scripts "https://github.com/beingmani/mbd-assignment/blob/main/Assign2/Ex1/ex1.ipynb"
    click py_scripts "https://github.com/beingmani/mbd-assignment/blob/main/Assign2/Ex1/ex1.py"
    click fr_src "https://github.com/beingmani/mbd-assignment/blob/main/Assign2/Ex3/src/com/mbdassign2/frdrecommender/FriendRecommender.java"
    click fr_tests "https://github.com/beingmani/mbd-assignment/tree/main/Assign2/Ex3/src/test/mbdassign/wordfreq"
    click hdfs_in3 "https://github.com/beingmani/mbd-assignment/tree/main/Assign2/Ex3/input"
    click hdfs_out3 "https://github.com/beingmani/mbd-assignment/tree/main/Assign2/Ex3/output"
    click mkdocs_cfg "https://github.com/beingmani/mbd-assignment/blob/main/Assign1/docs/mkdocs.yml"
    click docs_src "https://github.com/beingmani/mbd-assignment/tree/main/Assign1/docs/docs"
    click site "https://github.com/beingmani/mbd-assignment/tree/main/Assign1/docs/site"

    %% Styles
    classDef storage fill:#aaf,stroke:#000
    classDef compute fill:#fea,stroke:#000
    classDef dev fill:#afa,stroke:#000
    classDef docs fill:#ccc,stroke:#000
```
