How mimouni-hicham/als_hadoop works
Project type and purpose
• This is a Big-Data batch-processing pipeline that implements Alternating Least Squares (ALS) matrix factorization on a large input matrix M using Hadoop MapReduce.
• Goal: factorize a (10000×512) matrix M into two lower-rank matrices W and H (dimensions 10000×k and 512×k), iterating until convergence. Widely used for recommendation systems.
File structure
• Top-level:
• GenerateMatrix.java – standalone generator of the input matrix
• matrix.txt – sample matrix
• build.sh – compile script
• README.md – documentation
• src/ – contains all Hadoop Java code
– ALSDriver.java – driver/orchestrator
– MatrixLineMapper.java / PartialProductMapper.java – Map logic
– WCalculatorReducer.java / HCalculatorReducer.java – Reduce logic
– MatrixOperations.java – linear-algebra routines
– util/ – helper classes (vector.java, another MatrixOperations.java)
• output_examples/ – sample outputs for W and H
Key insights from README
• Three phases:
1. Generate or initialize W
2. Using fixed W + input M, compute H via MapReduce
3. Using fixed H + M, compute W via MapReduce
→ repeat for MAX_ITER
• Instructions for compiling, pushing to HDFS, running Hadoop jobs, and post-hoc Python evaluation.
a. Main components
• Input Generator: GenerateMatrix.java (local)
• Distributed Storage: HDFS clusters
• Orchestrator: ALSDriver (runs on master node)
• MapReduce Job #1 (Compute H):
• MatrixLineMapper → PartialProductMapper → shuffle → HCalculatorReducer
• MapReduce Job #2 (Compute W):
• MatrixLineMapper → PartialProductMapper → shuffle → WCalculatorReducer
• Utility library: MatrixOperations, vector (packed into job JAR)
• Post-processing/Evaluation: Python script (external)
b. Relationships & data-flow
1. GenerateMatrix.java → matrix.txt → Put into HDFS /user/matrix/
2. ALSDriver reads /user/matrix/ + current W (or initial random W), submits MR job to compute H → writes /user/output_h/
3. ALSDriver reads /user/output_h/, submits MR job to compute W → writes /user/output_w/
4. Loop back until MAX_ITER
5. Final outputs read by Python evaluation script
c. Architectural patterns & principles
• Batch processing via Hadoop MapReduce
• Driver-controlled iterative workflow (external loop implemented in ALSDriver.java)
• Separation of concerns: mapping transforms, partial-product mapping, reduction to solve least squares
• Reusable utilities abstract linear algebra
d. Key technologies
• Java 8+ (JDK)
• Hadoop MapReduce API
• HDFS for storage
• Bash (build.sh)
• Python/NumPy (evaluation)
Diagramming conventions
• Blue rectangles for storage (HDFS input/output directories)
• Green rectangles for compute components (mappers, reducers, driver)
• Gray rectangles for external scripts (GenerateMatrix, Python evaluator)