# mimouni-hicham/als_hadoop architecture

> Architecture diagram of the GitHub repository mimouni-hicham/als_hadoop, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mimouni-hicham/als_hadoop
- Repository: https://github.com/mimouni-hicham/als_hadoop

## Overview

##### Project type and purpose
- This is a Big-Data batch-processing pipeline that implements Alternating Least Squares (ALS) matrix factorization on a large input matrix `M` using Hadoop MapReduce.
- Goal: factorize a (10000×512) matrix `M` into two lower-rank matrices `W` and `H` (dimensions 10000×k and 512×k), iterating until convergence. Widely used for recommendation systems.

##### File structure
- Top-level:
- GenerateMatrix.java – standalone generator of the input matrix
- matrix.txt – sample matrix
- build.sh – compile script
- README.md – documentation
- src/ – contains all Hadoop Java code
– ALSDriver.java – driver/orchestrator
– MatrixLineMapper.java / PartialProductMapper.java – Map logic
– WCalculatorReducer.java / HCalculatorReducer.java – Reduce logic
– MatrixOperations.java – linear-algebra routines
– util/ – helper classes (vector.java, another MatrixOperations.java)
- output_examples/ – sample outputs for `W` and `H`

##### Key insights from README
- Three phases:
1. Generate or initialize `W`
2. Using fixed `W` + input `M`, compute `H` via MapReduce
3. Using fixed `H` + `M`, compute `W` via MapReduce
→ repeat for MAX_ITER
- Instructions for compiling, pushing to HDFS, running Hadoop jobs, and post-hoc Python evaluation.

a. Main components
- Input Generator: GenerateMatrix.java (local)
- Distributed Storage: HDFS clusters
- Orchestrator: ALSDriver (runs on master node)
- MapReduce Job #1 (Compute H):
- MatrixLineMapper → PartialProductMapper → shuffle → HCalculatorReducer
- MapReduce Job #2 (Compute W):
- MatrixLineMapper → PartialProductMapper → shuffle → WCalculatorReducer
- Utility library: MatrixOperations, vector (packed into job JAR)
- Post-processing/Evaluation: Python script (external)

b. Relationships & data-flow
1. GenerateMatrix.java → matrix.txt → Put into HDFS `/user/matrix/`
2. ALSDriver reads `/user/matrix/` + current `W` (or initial random `W`), submits MR job to compute `H` → writes `/user/output_h/`
3. ALSDriver reads `/user/output_h/`, submits MR job to compute `W` → writes `/user/output_w/`
4. Loop back until MAX_ITER
5. Final outputs read by Python evaluation script

c. Architectural patterns & principles
- Batch processing via Hadoop MapReduce
- Driver-controlled iterative workflow (external loop implemented in ALSDriver.java)
- Separation of concerns: mapping transforms, partial-product mapping, reduction to solve least squares
- Reusable utilities abstract linear algebra

d. Key technologies
- Java 8+ (JDK)
- Hadoop MapReduce API
- HDFS for storage
- Bash (build.sh)
- Python/NumPy (evaluation)

##### Diagramming conventions
- Blue rectangles for storage (HDFS input/output directories)
- Green rectangles for compute components (mappers, reducers, driver)
- Gray rectangles for external scripts (GenerateMatrix, Python evaluator)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Scripts
    GM["GenerateMatrix"]:::external
    Build["build.sh"]:::external
    README["README.md"]:::external

    %% Storage
    HDFS_IN["HDFS: /user/matrix"]:::storage
    HDFS_H["HDFS: /user/output_h"]:::storage
    HDFS_W["HDFS: /user/output_w"]:::storage

    %% Driver
    ALS["ALSDriver"]:::compute

    %% Compute H Subgraph
    subgraph "Compute H Job"
        HMap1["MatrixLineMapper"]:::compute
        PMap1["PartialProductMapper"]:::compute
        HRed["HCalculatorReducer"]:::compute
    end

    %% Compute W Subgraph
    subgraph "Compute W Job"
        HMap2["MatrixLineMapper"]:::compute
        PMap2["PartialProductMapper"]:::compute
        WRed["WCalculatorReducer"]:::compute
    end

    %% Utilities
    UtilCore["MatrixOperations (core)"]:::utility
    UtilUtil["MatrixOperations & vector (util)"]:::utility

    %% Python Evaluator
    Py["Python Evaluator"]:::external

    %% Data Flow
    GM -->|"generate matrix"| HDFS_IN
    Build --> GM
    README -.-> ALS
    README -.-> GM

    HDFS_IN -->|"reads matrix"| ALS

    ALS -->|"submits Compute H"| HMap1
    HMap1 --> PMap1 --> HRed -->|"writes H"| HDFS_H

    HDFS_H -->|"iter data"| ALS
    ALS -->|"submits Compute W"| HMap2
    HMap2 --> PMap2 --> WRed -->|"writes W"| HDFS_W

    HDFS_W -->|"final output"| Py
    HDFS_W -->|"next iter"| ALS

    %% Utility dependencies
    UtilCore -.-> HMap1
    UtilCore -.-> PMap1
    UtilCore -.-> HRed
    UtilCore -.-> HMap2
    UtilCore -.-> PMap2
    UtilCore -.-> WRed
    UtilUtil -.-> HMap1
    UtilUtil -.-> PMap1
    UtilUtil -.-> HRed
    UtilUtil -.-> HMap2
    UtilUtil -.-> PMap2
    UtilUtil -.-> WRed

    %% Click Events
    click GM "https://github.com/mimouni-hicham/als_hadoop/blob/main/GenerateMatrix.java"
    click ALS "https://github.com/mimouni-hicham/als_hadoop/blob/main/ALS_Factorization/src/ALSDriver.java"
    click HMap1 "https://github.com/mimouni-hicham/als_hadoop/blob/main/ALS_Factorization/src/MatrixLineMapper.java"
    click PMap1 "https://github.com/mimouni-hicham/als_hadoop/blob/main/ALS_Factorization/src/PartialProductMapper.java"
    click HRed "https://github.com/mimouni-hicham/als_hadoop/blob/main/ALS_Factorization/src/HCalculatorReducer.java"
    click HMap2 "https://github.com/mimouni-hicham/als_hadoop/blob/main/ALS_Factorization/src/MatrixLineMapper.java"
    click PMap2 "https://github.com/mimouni-hicham/als_hadoop/blob/main/ALS_Factorization/src/PartialProductMapper.java"
    click WRed "https://github.com/mimouni-hicham/als_hadoop/blob/main/ALS_Factorization/src/WCalculatorReducer.java"
    click UtilCore "https://github.com/mimouni-hicham/als_hadoop/blob/main/ALS_Factorization/src/MatrixOperations.java"
    click UtilUtil "https://github.com/mimouni-hicham/als_hadoop/blob/main/ALS_Factorization/src/util/MatrixOperations.java"
    click UtilUtil "https://github.com/mimouni-hicham/als_hadoop/blob/main/ALS_Factorization/src/util/vector.java"
    click Build "https://github.com/mimouni-hicham/als_hadoop/blob/main/build.sh"
    click README "https://github.com/mimouni-hicham/als_hadoop/blob/main/README.md"

    %% Styles
    classDef storage fill:#BBDEFB,stroke:#1E88E5,color:#0D47A1;
    classDef compute fill:#C8E6C9,stroke:#43A047,color:#1B5E20;
    classDef utility fill:#FFF9C4,stroke:#FDD835,color:#F57F17,stroke-dasharray: 5 5;
    classDef external fill:#E0E0E0,stroke:#9E9E9E,color:#424242;
```
