How doubong/customersegmentation-recommendationsystem works
File Structure
• Top‐level folders correspond to distinct pipeline stages:
– Data Preprocessing: Jupyter notebooks and CSVs for cleaning raw data
– HadoopStreaming: Python Mapper.py and Reducer.py using Hadoop Streaming
– Mapreduce Work: Java/Maven-based MapReduce (CustomerMapper, CustomerReducer, Driver)
– K-Means: Jupyter notebook applying clustering on aggregated output
• Deployment/config: pom.xml for Java build, target/*.jar for execution, hadoop-local result docs.
Extract Key Insights from README
• “Phát triển một hệ thống … bằng Hadoop trên Microsoft Azure.”
Main Components & Their Relationships
a. Data Sources & Storage
– Raw transaction CSV (“Online Retail.csv”) stored in Azure Blob Storage or Data Lake.
b. Data Preprocessing
– Jupyter notebooks (Python/Pandas) read raw CSV, produce cleaned_retail_data.csv.
– Output written back to Blob Storage (or HDFS on HDInsight).
c. MapReduce Layer
– Hadoop Streaming job (Mapper.py, Reducer.py) runs on HDInsight, produces intermediate aggregates (result_mapreduce.csv).
– Java MapReduce (App.java, CustomerMapper/CustomerReducer) processes cleaned data in a more scalable/prod manner.
– Both write outputs to HDFS/Azure Blob.
d. Clustering
– K-Means notebook loads MapReduce output, uses scikit-learn to assign customers to clusters, writes clustered_customers_done.csv.
e. Visualization & Reporting
– Power BI dashboards (Customer_Segmentation_Dashboard.pbix) connect to clustered output (via direct file or Azure SQL/Data Lake).
– Final PDF/PBIX for stakeholders.
f. (Optional) Recommendation Engine
– Future component that consumes clusters to generate personalized suggestions.
Architectural Patterns & Design Principles
• Batch data‐processing pipeline on Azure/HDInsight
• Modularity: each stage isolated (cleaning, streaming, Java MR, clustering, viz)
• Separation of concerns: compute jobs vs. notebooks vs. dashboards
• Scalability: Hadoop cluster for large-scale MapReduce
• Extensibility: can add new ML stages (recommendation) or swap Java for Spark.