# andrez00/hadoop-coursera architecture

> Architecture diagram of the GitHub repository andrez00/hadoop-coursera, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/andrez00/hadoop-coursera
- Repository: https://github.com/andrez00/hadoop-coursera

## Overview

##### Project type and purpose
- This is not a typical “product” codebase but a set of educational Hadoop and Spark exercises (MapReduce word count, joins, PySpark pipelines).
- The README (“# Hadoop”) and the folder names Week4_Lesson1/2 and Week5_Lesson1/2/3 reveal the purpose: teaching fundamentals of Hadoop MapReduce (mappers, reducers, intermediate joins) and Spark joins.

##### File structure
- Top-level: README.md, About the Course.txt, Week4_Lesson1, Week4_Lesson2, Week5_Lesson1/2/3.
- Week4 covers classic Hadoop MapReduce jobs:
- wordcount_mapper.py / wordcount_reducer.py
- join1_mapper.py / join1_reducer.py and join2_mapper.py / join2_reducer.py with synthetic data generators.
- Week5 shifts to PySpark IPython setups (install notes) and join examples.
- No “src” or “backend/frontend” separation, but clear separation by lesson folders.

##### README for additional insights
- README is minimal (“# Hadoop”), but folder names and text files contain instructions and outputs showing Hadoop command lines, expected outputs, and Spark code snippets.

##### Elements
– Client / Notebook
– HDFS NameNode & DataNodes
– YARN ResourceManager & NodeManagers
– MapReduce Tasks (Mapper, Reducer)
– Spark Driver & Executors
– Input / Output Data Stores
– From client → job submission → ResourceManager
– From DataNodes → mappers → shuffle → reducers → output
– From HDFS → Spark executors → HDFS
– Blue for storage components (NameNode, DataNodes)
– Green for compute (mappers, reducers, executors)
– Orange for the client/notebook interface

##### Detailed guidance
- On each DataNode, show mapper and reducer containers for Lesson1 & Lesson2 jobs.
- At the left, represent the user/client environment where wordcount_mapper.py, join scripts, and spark notebooks reside.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Environment
    subgraph "Client / Notebook" 
        direction TB
        WC_Mapper["WordCount Mapper"]:::compute
        WC_Reducer["WordCount Reducer"]:::compute
        Join1_Mapper["Join1 Mapper"]:::compute
        Join1_Reducer["Join1 Reducer"]:::compute
        Join2_Mapper["Join2 Mapper"]:::compute
        Join2_Reducer["Join2 Reducer"]:::compute
        Spark_Notebook["PySpark Notebook"]:::client
    end

    %% Hadoop Cluster
    subgraph "Hadoop Cluster"
        direction TB
        subgraph Storage
            direction TB
            NameNode["NameNode"]:::storage
            DataNodes["DataNodes"]:::storage
        end
        subgraph "YARN / MapReduce Framework"
            direction TB
            ResourceManager["ResourceManager"]:::compute
            NodeManagers["NodeManagers"]:::compute
        end
        subgraph "MapReduce Jobs" 
            direction TB
            Mapper_Container["Mapper Containers"]:::compute
            Reducer_Container["Reducer Containers"]:::compute
        end
    end

    %% Spark Cluster
    subgraph "Spark Cluster"
        direction TB
        Spark_Driver["Spark Driver"]:::compute
        Spark_Executors["Executors"]:::compute
    end

    %% Data Flow
    WC_Mapper -->|"submit WordCount job"| ResourceManager
    Join1_Mapper -->|"submit Join1 job"| ResourceManager
    Join2_Mapper -->|"submit Join2 job"| ResourceManager
    Spark_Notebook -->|"spark-submit"| Spark_Driver

    ResourceManager -->|"allocate containers"| NodeManagers
    NodeManagers -->|"launch"| Mapper_Container
    Mapper_Container -->|"map() on HDFS data"| Reducer_Container
    Reducer_Container -->|"reduce() & write()"| DataNodes

    NameNode -->|"store input data"| DataNodes
    DataNodes -->|"serve data"| Spark_Executors
    Spark_Driver -->|"schedule tasks"| Spark_Executors
    Spark_Executors -->|"transform & join data"| DataNodes

    %% Click Events for component mappings
    click WC_Mapper "https://github.com/andrez00/hadoop-coursera/blob/master/Week4_Lesson1/wordcount_mapper.py"
    click WC_Reducer "https://github.com/andrez00/hadoop-coursera/blob/master/Week4_Lesson1/wordcount_reducer.py"
    click Join1_Mapper "https://github.com/andrez00/hadoop-coursera/blob/master/Week4_Lesson2/join1_mapper.py"
    click Join1_Reducer "https://github.com/andrez00/hadoop-coursera/blob/master/Week4_Lesson2/join1_reducer.py"
    click Join2_Mapper "https://github.com/andrez00/hadoop-coursera/blob/master/Week4_Lesson2/join2_mapper.py"
    click Join2_Reducer "https://github.com/andrez00/hadoop-coursera/blob/master/Week4_Lesson2/join2_reducer.py"
    click Spark_Notebook "https://github.com/andrez00/hadoop-coursera/blob/master/Week5_Lesson1/Install PySpark IPython.txt"

    %% Styles
    classDef storage fill:#DEEBFF,stroke:#3182CE,color:#0B3D91
    classDef compute fill:#E6F4EA,stroke:#31A354,color:#006B2C
    classDef client fill:#FFF4E6,stroke:#FF7F0E,color:#D35400
```
